Searching for Entities returns inconsistent results #216

b4dpxl · 2019-09-13T09:14:15Z

Description

When trying to search for an entity to add to a report, the search results are inconsistent. For example, trying to add "Input Prompt" as a TTP:

search for input does not match
search for input prompt does not match
search for prompt does match

Environment

OS (where OpenCTI server runs): Ubuntu 18.04.3 LTS
OpenCTI version: OpenCTI 1.1.2
OpenCTI client: frontend
Other environment details: Running in Docker

Reproducible Steps

Steps to create the smallest reproducible scenario:

Browse to Report->Entities
Click the + button
Enter input into the search field

Expected Output

Input Prompt to appear under the TTP results section

Actual Output

It is not in the list

Additional information

It appears to be matching on the description only, not the name. This makes finding Entities to add via the search harder, as you need to know the exact match, and you can't even copy/paste the Entity name from another browser tab.

The text was updated successfully, but these errors were encountered:

richard-julien · 2019-09-14T09:45:29Z

Hi @b4dpxl , can you try to reproduce on https://demo.opencti.io? I try myself on the platform and i can see the TTP Input Prompt for all the cases. Thanks.

richard-julien · 2019-09-14T09:52:39Z

If it work, you can try to do this:

Edit the entity "Input Prompt"
Edit any field of the entity and save it
Retry the search

One more question. Do you start fresh on 1.1.2 or its an older platform migrated?

b4dpxl · 2019-09-14T12:10:47Z

Hi @richard-julien, thanks for the quick response. I agree, this works on the demo platform, but not on my deployment. This was migrated from 1.1.1 (well, I updated the docker-compose.yml but kept the persistent storage from 1.1.1).

I also tried editing the Input Prompt entity, which made no difference. Searching for variations of "Input" and "Input Prompt", in different cases, returns at most 6 TTP results on my deployment. Demo returns 18 results for "Input Prompt". Comparing the entities in the front end, they appear the same (imported from Mitre).

I wonder if this is somehow related to the issue I raised here: OpenCTI-Platform/client-python#21 ?

richard-julien · 2019-09-14T13:21:18Z

I think the client-python#21 is the same problem yes. The python lib just use the graphQL api, so you will have the same problem.

Can you execute the docker ps command and copy/paste the result?
Same thing for docker images?

Can you also configure your platform with logs_level = debug, restart the container and copy/paste the docker logs result when you try to update the "Input Prompt" TTP?

Thanks.

b4dpxl · 2019-09-16T07:20:05Z

docker ps

95b04d314884        opencti/worker:1.1.2                                  "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_worker-import_3
dba7e24b5de2        opencti/worker:1.1.2                                  "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_worker-import_2
93d67d9e450e        opencti/worker:1.1.2                                  "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_worker-export_1
86a5c49d205d        opencti/connector-opencti:1.1.2                       "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_connector-opencti_1
a2e0fd96e5c4        opencti/worker:1.1.2                                  "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_worker-import_1
2d9263152860        opencti/worker:1.1.2                                  "/entrypoint.sh"         4 days ago          Up 4 days                                                                                                        docker_worker-import_4
ec33c1596bdb        opencti/platform:1.1.2                                "/entrypoint.sh"         4 days ago          Up 4 days           4000/tcp, 0.0.0.0:8080->8080/tcp                                                             docker_opencti_1
f6416f1afb9b        rabbitmq:3.7.17-management                            "docker-entrypoint.s…"   4 days ago          Up 4 days           4369/tcp, 5671/tcp, 0.0.0.0:5672->5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp   docker_rabbitmq_1
9702306658cb        docker.elastic.co/elasticsearch/elasticsearch:7.3.0   "/usr/local/bin/dock…"   4 days ago          Up 4 days           9200/tcp, 9300/tcp                                                                           docker_elasticsearch_1
22e0282b0c34        graknlabs/grakn:1.5.8                                 "./grakn-docker.sh"      4 days ago          Up 4 days           0.0.0.0:48555->48555/tcp                                                                     docker_grakn_1
29758f4dc7d8        redis:5.0.5                                           "docker-entrypoint.s…"   4 days ago          Up 4 days           6379/tcp                                                                                     docker_redis_1
652fd0eae9df        opencti/connector-cve:1.1.2                           "/entrypoint.sh"         2 weeks ago         Up 2 weeks                                                                                                       cve_connector-cve_1

docker images

opencti/worker                                  1.1.2               7dd79ac2bf85        10 days ago         552MB
opencti/platform                                1.1.2               0e751b68ab61        10 days ago         1.59GB
rabbitmq                                        3.7.17-management   7601e834fa14        4 weeks ago         177MB
redis                                           5.0.5               f7302e4ab3a8        4 weeks ago         98.2MB
opencti/worker                                  1.1.1               deebad3ab542        6 weeks ago         544MB
opencti/platform                                1.1.1               6ba2d445f688        6 weeks ago         1.58GB
opencti/connector-cve                           1.1.2               db0033d8b8b3        6 weeks ago         507MB
opencti/connector-opencti                       1.1.2               5d469f9480ad        6 weeks ago         507MB
opencti/connector-mitre                         1.1.2               1960ba41fe80        6 weeks ago         507MB
docker.elastic.co/elasticsearch/elasticsearch   7.3.0               bdaab402b220        7 weeks ago         806MB
graknlabs/grakn                                 1.5.8               08d0e7e15029        49 years ago        689MB
graknlabs/grakn                                 1.5.7               47b3bd9f931c        49 years ago        688MB

Adding --log-level=debug to docker-compose or changing the OPENCTI_LOG_LEVEL in docker-compose.yml made no difference to the output when changing the TTP.

I can try recreating the whole thing from scratch, but last time it took 5+ days to import the mitre feed. Incidentally, I bodged together a hack for that import, to only import entries modified since the import last run, if you're interested. It will only work if using a config file though.

richard-julien · 2019-09-16T08:55:42Z

Sorry, my fault. The docker option is APP__LOGS_LEVEL=debug
I understand you don't want to reinject everything from scratch.
The logs should help to understand whats going on platform side.
Then the next step will be to dig in to elasticsearch to see the index definition. Its look like a bug occurs during the index definition for the 1.1.2 upgrade but I cant say exactly what and why....

For your import hack to have a state for the MITR, yes it could be interesting to see your approach. We currently think about the best solution to introduce state in connectors. Dont hesitate to submit a PR in the MITR connector.

b4dpxl · 2019-09-16T09:57:12Z

Thanks. Log file attached. I've also created a PR.
octi_debug.txt

This covers adding text to the TTP description and also changing the name (from "Input Prompt" to "Input Prompts"). Neither seemed to affect the searching. It was matched on searching for "prompts" (which is only in the title), but not "input" or "input prompts".

Thanks again

richard-julien · 2019-09-16T13:21:52Z

Your log explicitly mention that the platform reindex the TTP.

opencti_1            | debug: [GRAKN - infer: false] updateAttribute - insert > match $m has internal_id "49fcbfd8-e566-4642-a4e2-5734579904b8"; insert $m has name "Input Prompts";
opencti_1            | debug: [GRAKN - infer: false] getAttributes > Attack-Pattern V10178696
opencti_1            | debug: [ELASTICSEARCH] getAttributes get > Attack-Pattern V10178696 on stix_domain_entities
opencti_1            | debug: [ELASTICSEARCH] index > attack-pattern 49fcbfd8-e566-4642-a4e2-5734579904b8 in stix_domain_entities

Looks like the problem is inside the Elasticsearch stix_domain_entities index.
Can you execute this command and copy paste the result.

curl http://localhost:9200/stix_domain_entities | json_pp

Thanks

b4dpxl · 2019-09-17T09:11:24Z

Hi. Output attached. I'm not an ES expert, but it looks like name and alias are of keyword type, which I believe requires an exact match? Though the normalizer is making them lowercase, so that shouldn't be related to my other issue (assuming it's using ES not Grakn).

Thanks again.

octi_es_index.txt

richard-julien · 2019-09-17T11:48:20Z

Thanks for the extract, everything seems ok. The normalizer make them lowercase but the default analyzer split on whitespace, so it should work. Looking deeply in the code I see a potential problem to find elements like InputSuffix but nothing related directly to your problem. Just to bypass openCTI completly, can you try this command :

curl -H "Content-Type: application/json" -XGET 'http://localhost:9200/stix_domain_entities/_search' -d '{
  "query": { 
    "bool": {
          "must": [
            {
              "query_string": {
                "query": "*Input prompt*",
                "analyze_wildcard": true,
                "default_field": "*"
              }
            }
          ]
    }
  }
}' | json_pp

And see if you have all the TTP in the answer? Thanks

b4dpxl · 2019-09-17T12:09:44Z

That query returns the expected TTPs when looking for *input prompt* but nothing on *input* or *prompt*. However, prompt and input (no wildcards) work as expected.

Interestingly, *prompt* returns the "Input Prompt Mitigation" Course of Action, but *input* doesn't.

richard-julien · 2019-09-17T12:29:14Z

Ok, can you try to add explain=true to the different queries and upload the result?

curl -H "Content-Type: application/json" -XGET 'http://localhost:9200/stix_domain_entities/_search?explain=true' -d '{
  "query": { 
    "bool": {
          "must": [
            {
              "query_string": {
                "query": "*Input prompt*",
                "analyze_wildcard": true,
                "default_field": "*"
              }
            }
          ]
    }
  }
}' | json_pp

Thanks

b4dpxl · 2019-09-17T12:47:23Z

Thanks. I hope the attached helps. The individual files are named after the search, e.g. star_prompt_star.txt for *prompt*.
explained.zip

SamuelHassine · 2019-10-25T10:30:48Z

@b4dpxl: this should now be fixed in 2.0.1.

richard-julien added the bug use for describing something not working as expected label Sep 14, 2019

SamuelHassine closed this as completed Oct 25, 2019

SamuelHassine added this to the Release 2.0.1 milestone Oct 25, 2019

SamuelHassine self-assigned this Oct 25, 2019

SamuelHassine added the solved use to identify issue that has been solved (must be linked to the solving PR) label Oct 25, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Searching for Entities returns inconsistent results #216

Searching for Entities returns inconsistent results #216

Searching for Entities returns inconsistent results #216

Searching for Entities returns inconsistent results #216

Comments

Description

Environment

Reproducible Steps

Expected Output

Actual Output

Additional information