Ianhelle/mp pivot phase3 2021 02 22 (microsoft#140)

* Typo in opening sentence

* Adding hash_account as separate item type to data_obfus.py

Making hash_ip more flexible - ignoring things like localhost
Updating documentation, tests and mapping file.
Correcting typo in timeline.py.

* Adding missed documentation for hash_account

* Initial code for Mordor driver and browser

* Mordor data provider and browser.

unit tests and documentation

* Fixing some linting errors.

* Fixed a couple of broken tests because of data providers API change.

* Replacing custom json reader with pd.read_json()

Added ability to set query defaults (like cache directory) from provider.
Fixed a bug in path construction for download file.
Clarified the description of the search functionality and corrected Mitre Attack => ATT&CK
Add URL for Mitre
Updated notebook and doc to reflect these changes.

* Fixing lint/formatting errors in vtlookupv3.

Some other random black reformatting
Added test_mordor_browser.py for notebook test.

* Updated formatting for new black version

* Updating pre-commit version

* Bug fix and nasty workaround for old test setup removed in pkg_config.py

* Update MordorData.rst doc with better intro section

* Splitting entities into separate modules

* Moved entities to datamodel package and initial refactoring for pivoting

* Renaming files to lowercase phase 1

* Renaming entities phase 2

* Start of pivot main library

* Commit to re-merge with master

* Code complete - still docs to do.

* Added test case and fix for couple of misc methods in Pivot and Entity

* Phase 1 code complete with docs.

* Fixing the credscan suppression for test_splunk_uploader

* Adding pre-release version, removing old config file.

* Initial dependency separation

* Implemented extras for msticpy install.

Refactored a few classes to make it easier to import and use modules if only partial msticpy install. Installing
Main one is data_providers - dynamically loading drivers. Also eventcluster and auditdextract.
Moved latter two into analysis folder.
Remove unneeded code from keyvault_client.py since Pete's code eliminated the need for them.
Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
Fixed up several unit tests to handle partial installs and still produce results (most should be skipped now instead of erroring).
Fixed a random bugs (like GeoIP Maxmind download)
Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
Added documentation to Installing.rst
Fixed some problems and renamed module locations in notebooks and RST docs.

* Additions/corrections to Installing.rst

* Somehow these two data files were changed.

* Bandit exception to except: pass

* Correction to FoliumMap.ipynb

* Removing dropna from read_csv in FoliumMap.ipynb

* Adding requirements-all and pre-commit hook to generate this file

* Adding vt, vt_graph to Sphinx mock list

* Added pivot_browser UI - pivot_browser.py

Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py
Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
Exposing get_timespan function in Pivot class as public function - in pivot.py.
Added Dns entity to several pivot functions - mp_pivot_reg.yaml

* Fixing some queries for more consistency.

Pivot data query functions now prefixed with table name.
Added ability for pivot functions to return raw output.
Added pyperclip to pkg dependencies exceptions.

* Some corrections to documentation in AzureSentinel and DataAcquisition docs.

Added lru_cache for geoip lookups.

* Fixing mordor tests and updating azure-mgmt-monitor version in setup.py extras

* PR updates adding comments, some grammer fixes and obfuscation of names.

* PR updates adding comments, some grammar fixes and obfuscation of names.

* Merge tag 'v0.9.0' into ianhelle/MP-Pivot-Phase2-2021-01-04

Fixing some test and linting errors after merge.
Removing lru_cache from ip_lookup in geoip.py

* Add joins for pivot data queries in pivot_data_queries.py

Add "print" query debug parameter in data_providers.py
Add find_entity function in entities __init__.py
Add alias "pivots" for get_pivot_list in entity.py
Add ability to set timespan more flexibly. Calling set_timespan no longer resets the timespan. Add PivotBrowser method to Pivot class - in pivot.py
Add missing entity list box in pivot_browser.py.
Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types.
Add positional params to pipeline step and cleaned up code in pivot_pipeline.py
Updated PivotFunctions.rst and PivotFunctions.ipynb for new functionality.
More tests for test_pivot.py (timespan)
New tests for PivotBrowser - test_pivot_browser.py
Enable and fix tests for pivot data query joins in test_pivot_data_queries_run.py
Add test for positional params in test_pivot_pipeline.py

* Suppressing expected user warnings in tests.

Fixing a bug with the "print_query" debug option being called from TIProviders/kql_base.py.
Cleaning up mordor data file cleanup in test_mordor_driver.py.
Adding an optimistic random delay to geoip.py to avoid instances in different processes trying to download the same file simultaneously. Really only an issue in multi-processing distributed tests.

* Bandit warning on use of random.randint()

Updating version

* Black version mismatch

* Adding notice and badge to Readme

* Small changes from PR review

* Adding column filtering accessor for pipeline

* tilookup functions to allow enabling/disabling individual providers

Fixed a problem with logon failure query parameters.

* Minor corrections to queries in kql_sent_az_network and kql_sent_azure

* Adding shortcut functions to entities

* Pivot phase 3 code complete with better joins, more pd accessor functions and short/friendly aliases for

pivot functions (especially queries)

* Updated documentation.

Added remove_pivot_funcs method to pivot.pivot.py + unit test in test_pivot
Extracted dup code from pivot_data_queries to pivot_register.
Fixing test in test_pivot_pd_accessor for renamed parameter.
Updating notebooks for some pivot function renaming.

* Spelling errors in SettingsEditor.rst

* Prospector doc string warnings fixed

* Add list_to_rows to pivot_pd_accessor.py

* Add list_to_rows and parse_json to pivot_pd_accessor.py

Adding str and repr to Edge in entity_graph.py
Fixing path handling in file.py
Adding (not yet used) graph_property.py to handle automatic creation of edges
Add better timestamp conversion in vtlookup3.py.
Remove unnecessary list comprehension in test_security_alert.py
Add tests for  list_to_rows and parse_json in pivot_pd_accessor.py - test_pivot_pd_accessor.py

* Merge PivotFunctions-Introduction notebook.

* Added missing entities: iot_device, mail_cluster, mail_message, mailbox, submission_mail

Added check for attribute name similarity in query_container
Fixed some errors in entities with description_str and identity fields
Add missing FullName property to Host
Fixed exception in IpAddress
Removing duplicate mp_config file
Fixing some test errors in test_mordor_driver (due to test file race conditions)

* Updating blog_articles with recent publications

Updating version to 1.1.0
Capturing output from load_user_defaults in nbinit.py
Correcting problem with passing namespace argument to notebooklets in user_config.py
Fixes to analyze_imports.py and import_analyzer.py that prevented use from commandline and use with other packages.

* Fix to list_to_rows pd accessor in pivot_pd_accessor.

* Changed timeline.py so that it doesn't error when no source_columns are supplied

* Fixing bug in calculating period in timespan.py

Adding find_anomaly_periods function and shorter alias for timeseries_anomalies_stl in timeseries.py
Cosmetic fixes in eventcluster.py

* Fixing an issue in ti_browser if passed an empty dataframe.

Updating requirements files for updated versions of lxml and aiohttp for security issues.

Fixing an error with changes in mitre attack website - switched to us…

…ing MITRE JSON github file. (microsoft#156)

Adding other files required for pip install --binary to run without wheel - README.md and requirements-dev.txt in MANIFEST.in

Ianhelle/msticpy1 0 0rc4 2021 04 05 (microsoft#150)

* Changed URL formatting so exception URLs use about="_blank" to open in new tab - in exceptions.py. Also switched to using a list for output instead of concatenated string in MsticpyUserError::_repr_html_()

Sourcery change in utility.py
Tidied up formatting of text and updated exception URL in wsconfig.py
nbinit.py:
- Add about="_blank" to URLs in warnings,
- Tidied up some text
- Changed red error to orange warning
- Trapped seaborn import error if not installed
- Print out list of imported packages by default
- Filtering warnings from call to WorkspaceConfig.py

* Added additional extra for KqlmagicCustom to install pyperclip

Bump version to rc5
az_connect defaults to all auth_methods if None supplied in params or config - azure_auth.py
azure_auth_core.py - added default_auth_methods() function
keyvault_settings.py - default to using all auth_methods
provider_settings.py - do not instantiate SecretSettings unless KeyVault config has settings
check_and_install_missing_packages - run "python -m pip" rather than just pip
wsconfig.py:
- Changed wording of wanings/error messages
- Does not now raise an exception if no suitable config found
- Does not raise a Python warning if we rely on searching to find a config file.
Changing default entity graph plot size in draw_entity_alert_graph. plot_entity_graph shows the plot by default unless hide=True parameter - nbdisplay.py
nbinit.py
- Changed all output so that it is HTML text vs. print
- Captured output from called functions that print output
- Minor rewording and changing header size of titles.
security_alert_graph.py - check if File entity has FullPath attribute before accessing it.
geoip.py - updating wording in exceptions if no API keyvault_settings
tilookup.py - added raise Config exception with help URIs if no providers are enabled - in lookup_ioc, lookup_iocs
Added moz_sql_parser back to mypy.ini
Changed test_pkg_config.py so that it gives clearer message when test fails - added "KqlmagicCustom[jupyter-extended]" to conda exceptions
Updated test_pkg_config.py to account for WorkspaceConfig no longer produces and error.

* Missed capturing warning output from WorkspaceConfig in nbinit.py

* If any warning from WorkspaceConfig, print this out - nbinit.py

* If running in IPython environment check_and_install_missing_packages() will using "pip" magic rather than subprocess in utility.py

Suppressing warnings in Kqlmagic load in kql_driver.py
File entity "FullPath" generation now handles None values for directory and separator more gracefully in file.py
Changed color attribute extraction so that it defaults to a color even if the node has no color attribute - in nbdisplay.py
Handle inter-entity references in child entities in security_alert.py
Handle no "Name" attribute in account entities.
Added additional test cases for nested entity references in test_security_alert.py

* Linux bug - Error loading secretsettings in env that doesn't support Keyring. Fixed this to test Keyring before trying to load. Also fixed logic to honor settings (subject to above check) - previously the param defaulted to True so would also load even if settings were = False.

* Minor Fixes

* Updating version to 1.0.0

Fixing test_security_alert.py unit test
Adding small test for KeyringClient - test_provider_secrets.py

* roll back DF boolean changes.

* Adding MC0001 Mcabe suppression to nbinit.py # 400

Fixing typo in DF name in syslog_utils.py

* Handling cases where empty data set is passed to timeline functions.

Fixing occasional race condition in testing with KeyringClient.is_keyring_available

* Updating KqlmagicCustom version

Sorcery fixes in syslog_utils.py

Co-authored-by: Pete Bryan <peter.bryan@microsoft.com>

Ianhelle/rel1.0.0 cleanup 2021 03 12 (microsoft#147)

* Cleanup and bug fixes for v1.0 - dependencies.

- Updated msticpy notebooks
- Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version
- Fix to test_nbinit.py and import_analyzer.py
- Added test script for dependencies/extras
- Updating version to pre3/rc3
- Fix to README
- Updated notebooksamples.rst with latest notebooks.notebooksamples
- Removed random unicode char in PivotFunctions.rst
- Added Releases.rst - pointer to GitHub release page.

* Fixing test error in test_nbinit.py

Some fixes to notebooks (mainly getting rid of uneeded Seaborn refs)

* Doc string error - prospector - in test_mp_extras.py

* More updates for release:

- Added links to medium articles in ReadtheDocs
- Fixed QueryProviderDocumenter.ipynb
- Generated new DataQueries.rst doc
- Added KqlmagicCustom[jupyter-basic] back to core components
- Updating docs for this in Installing.rst
- Updated test_pkg_imports.py to account for this.
- Setting KQLMAGIC_EXTRAS_REQUIRE env var in package __init__.py to prevent warnings in Kqlmagic
- Minor clarification in MPSettingsEditor.ipynb
- Reordered params in wsconfig.py WorkspaceConfig so that you can supply the workspace name as single positional parameter
- Change kql_driver.py so that you can supply a WorkspaceConfig instance as the "connection_str"
- Change to pivot_register to return single "raw" result if it is a list of one item
- Add Pivot and entities as auto-imported items
-

* Add create static method to entity to instantiate entity from dict or pd.Series

Fix bug in ip_utils.py convert to entities

* Fix to provider name handling in user_config.py

* Adding more queries for notebooklets - VMComputer and DNSEvents.

Fixing wording in SelectAlert widget
Fixing potential None value error in GetText and GetEnvironmentText
Fixing a few bugs and tidying in convert_to_ip_entities in ip_utils.py
Random linting errors.
Fixing issue microsoft#146 - Error is thrown when AzureSentinel config is not present in msticpyconfig.yaml file
Added extra tests for nbinit.py in test_nbinit.py

* Fixing infinite recursion issue in process_tree_utils.py
Issue microsoft#148

* Adding file lock to unit_test_lib::custom_mp_config - because settings are global, multiple test processes can change the settings on each other.

Minor updates to test_nbinit.py
Fix in pivot_register.py - _iterate_func did not pass through **kwargs to function. Also added a few comments to explain what's going on
Bug in security_alert_graph - if NTDomain attribute is None
Added local function cache to better handle repeated IPs
Missing update to all_ips perf optimizatio nin convert_to_ip_entities
Adding filelock to dev requirements-dev.txt

* Corrected pip extras syntax in Installing.rst

Invalid return type in pkg_config.py:: validate_config
Added prompt_for_ws function to wsconfig.py. Also added more detail to warnings on how to fix things.
Replace matplotlib draw_entity_alert_graph with Bokeh version in nbdisplay.py
nbinit.py:
- Added resource URLs to warnings
- added KQLMAGIC_CONFIGURATION to enable trying AzureCLI SSO by default.
- added pandas config to return schema with dataframe html to render using native nteract data browser.

* Adding markdown to requirements-dev.txt

* black formatting of security_alert

* Adding beautifulsoup4 and markdown to dev/test requirements in requirements-dev.txt and conda-reqs-dev.txt

Adding time unit control to QueryTime widget to allow interactive setting of day/week/month. Also increased the max range for these units - in nbwidgets.py

* Fixing test error - seems like subtle change or bug in pandas groupby behavior - in sessionize.py

Changing Development Status classifier and adding some extra keywords in setup.py
Adding more verbose output to try to catch spurious errors in test_nbinit.py

* Workarounds for some test errors in test_nbinit.py and test_user_config.py

Updating requirements-dev.txt with pip-compatible versions.
Adding same updates to conda-reqs-dev.txt and conda-reqs-dev-pip.txt

Cleanup and bug fixes for v1.0 - dependencies. (microsoft#142)

* Cleanup and bug fixes for v1.0 - dependencies.

- Updated msticpy notebooks
- Updated setup.py, requirements.txt and conda-reqs with new/non-conflicting version
- Fix to test_nbinit.py and import_analyzer.py
- Added test script for dependencies/extras
- Updating version to pre3/rc3
- Fix to README
- Updated notebooksamples.rst with latest notebooks.notebooksamples
- Removed random unicode char in PivotFunctions.rst
- Added Releases.rst - pointer to GitHub release page.

* Fixing test error in test_nbinit.py

Some fixes to notebooks (mainly getting rid of uneeded Seaborn refs)

* Doc string error - prospector - in test_mp_extras.py

Update version to 1.0.0.pre2

Ianhelle/settings mgmt 2021 02 02 (microsoft#136)

* Typo in opening sentence

* Adding hash_account as separate item type to data_obfus.py

Making hash_ip more flexible - ignoring things like localhost
Updating documentation, tests and mapping file.
Correcting typo in timeline.py.

* Adding missed documentation for hash_account

* Initial code for Mordor driver and browser

* Mordor data provider and browser.

unit tests and documentation

* Fixing some linting errors.

* Fixed a couple of broken tests because of data providers API change.

* Replacing custom json reader with pd.read_json()

Added ability to set query defaults (like cache directory) from provider.
Fixed a bug in path construction for download file.
Clarified the description of the search functionality and corrected Mitre Attack => ATT&CK
Add URL for Mitre
Updated notebook and doc to reflect these changes.

* Fixing lint/formatting errors in vtlookupv3.

Some other random black reformatting
Added test_mordor_browser.py for notebook test.

* Updated formatting for new black version

* Updating pre-commit version

* Bug fix and nasty workaround for old test setup removed in pkg_config.py

* Update MordorData.rst doc with better intro section

* Splitting entities into separate modules

* Moved entities to datamodel package and initial refactoring for pivoting

* Renaming files to lowercase phase 1

* Renaming entities phase 2

* Start of pivot main library

* Commit to re-merge with master

* Code complete - still docs to do.

* Added test case and fix for couple of misc methods in Pivot and Entity

* Phase 1 code complete with docs.

* Fixing the credscan suppression for test_splunk_uploader

* Adding pre-release version, removing old config file.

* Initial dependency separation

* Implemented extras for msticpy install.

Refactored a few classes to make it easier to import and use modules if only partial msticpy install. Installing
Main one is data_providers - dynamically loading drivers. Also eventcluster and auditdextract.
Moved latter two into analysis folder.
Remove unneeded code from keyvault_client.py since Pete's code eliminated the need for them.
Made AzureSentinel and MDE the preferred names for LogAnalytics and MDE drivers.
Fixed up several unit tests to handle partial installs and still produce results (most should be skipped now instead of erroring).
Fixed a random bugs (like GeoIP Maxmind download)
Fixed pivot_register_reader to skip classes that cannot be instantiated (e.g. IPStack if user doesn't have API key)
Added documentation to Installing.rst
Fixed some problems and renamed module locations in notebooks and RST docs.

* Additions/corrections to Installing.rst

* Somehow these two data files were changed.

* Bandit exception to except: pass

* Correction to FoliumMap.ipynb

* Removing dropna from read_csv in FoliumMap.ipynb

* Adding requirements-all and pre-commit hook to generate this file

* Adding vt, vt_graph to Sphinx mock list

* Added pivot_browser UI - pivot_browser.py

Added ability to read pipeline definitions from yaml files - pivot_pipeline.py
Adding pivot.tee_exec pipeline function - in pivot_pd_accessor.py
Add ability to add arbitrary/ad hoc functions as pivots - in pivot.py
Exposing get_timespan function in Pivot class as public function - in pivot.py.
Added Dns entity to several pivot functions - mp_pivot_reg.yaml

* Fixing some queries for more consistency.

Pivot data query functions now prefixed with table name.
Added ability for pivot functions to return raw output.
Added pyperclip to pkg dependencies exceptions.

* Some corrections to documentation in AzureSentinel and DataAcquisition docs.

Added lru_cache for geoip lookups.

* User environment configuration for notebooks.

Added minimal output from nbinit to show imported modules (I'd noticed some examples of people import stuff that had already been imported)

* Fixing mordor tests and updating azure-mgmt-monitor version in setup.py extras

* Some fixes and changes to the UserDefaults feature - esp the format of the config settings.

Also
- some fixes to tests for test_pkg_imports and import_analyzer.py
- fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets
- fix for ipwidgets warning about deprecated on_submit() method
- multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
- added SplunkProvider.rst doc for Splunk provider
- fixed issue in nbinit.py where extra_imports were being lost.
- fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
- fixed several issues in test_mp_release.cmd with messed up folders/current folder.

* MSTICPY config settings management
Two main classed
- MpConfigFile (to manage settings file and do a few utility things)
- MpConfigEdit (to edit settings for mp config sections)
Still to add docs/notebook

* PR updates adding comments, some grammer fixes and obfuscation of names.

* PR updates adding comments, some grammar fixes and obfuscation of names.

* Some fixes and changes to the UserDefaults feature - esp the format of the config settings.

Also
- some fixes to tests for test_pkg_imports and import_analyzer.py
- fix to config2kv.py to correct some problems, Also added a function to retrieve and show current KV secrets
- fix for ipwidgets warning about deprecated on_submit() method
- multiple fixes for typos and duplicate section names in: DataProviders.rst, UploadData.rst, PivotFunctions.rst
- added SplunkProvider.rst doc for Splunk provider
- fixed issue in nbinit.py where extra_imports were being lost.
- fix for QueryTime in nbwidgets.py - exception if user types invalid value into date field.
- fixed several issues in test_mp_release.cmd with messed up folders/current folder.

MSTICPY config settings management
Two main classed
- MpConfigFile (to manage settings file and do a few utility things)
- MpConfigEdit (to edit settings for mp config sections)
Still to add docs/notebook

Additional tests, start of validation checks

Added validation step (none-blocking) to forms

Fixing some settings validation issues
Fixed default values being overwritten for new items

Adding more tests and fixes.

Added check_version.py and added call to this from nbinit.py
Added Mordor and LocalData as configurable providers in settings.
mordor_driver, local_data_driver and azure_auth now check settings for defaults.
Add list() type to mpconfig_defaults.yaml

Settings documentation and notebook.

Also updating README.md and PackageSummary.rst with something more contemporary.

* Some tests failing after merge.

Fixed URL in README.md

* Merge tag 'v0.9.0' into ianhelle/MP-Pivot-Phase2-2021-01-04

Fixing some test and linting errors after merge.
Removing lru_cache from ip_lookup in geoip.py

* test_file_browsert test failing because it was trying to change into parent folder and parent folder doesn't exist in CI test environment

* Add joins for pivot data queries in pivot_data_queries.py

Add "print" query debug parameter in data_providers.py
Add find_entity function in entities __init__.py
Add alias "pivots" for get_pivot_list in entity.py
Add ability to set timespan more flexibly. Calling set_timespan no longer resets the timespan. Add PivotBrowser method to Pivot class - in pivot.py
Add missing entity list box in pivot_browser.py.
Switched engine to "Python" for pd.read_csv in pivot_magic_core.py to handle more formatting types.
Add positional params to pipeline step and cleaned up code in pivot_pipeline.py
Updated PivotFunctions.rst and PivotFunctions.ipynb for new functionality.
More tests for test_pivot.py (timespan)
New tests for PivotBrowser - test_pivot_browser.py
Enable and fix tests for pivot data query joins in test_pivot_data_queries_run.py
Add test for positional params in test_pivot_pipeline.py

* Suppressing expected user warnings in tests.

Fixing a bug with the "print_query" debug option being called from TIProviders/kql_base.py.
Cleaning up mordor data file cleanup in test_mordor_driver.py.
Adding an optimistic random delay to geoip.py to avoid instances in different processes trying to download the same file simultaneously. Really only an issue in multi-processing distributed tests.

* Fixing test error in test_user_config.py

McCabe complexity warning in config2kv.py

* Updating version

* Bandit warning on use of random.randint()

Updating version

* Removing fake secret from MPSettingsEditor.ipynb
Moving list definition for mypy in local_data_driver.py
Black reformatting test_user_config.py

* Failing test and linter warnings

* Adding notice and badge to Readme

* Adding documentation diagrams

* Updates from PR.

Also fixing a bug and merge conflict in mp_config_file.py where I was passing the whole URL as the secret name. Also put a catch for this in keyvault_client.py.

Updating version and adding compatibility fix so we can work with dns…

…python < 2.0.0 (microsoft#135)

(dnspython 2.x is incompatible with ipwhois)

Bump azure-identity from 1.4.0 to 1.5.0 (microsoft#129)

* Bump azure-identity from 1.4.0 to 1.5.0

Bumps [azure-identity](https://github.com/Azure/azure-sdk-for-python) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/Azure/azure-sdk-for-python/releases)
- [Commits](Azure/azure-sdk-for-python@azure-identity_1.4.0...azure-identity_1.5.0)

Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

* Update conda-reqs-pip.txt

Change azure-identity to >=1.5.0

* Change azure-identity to >=1.5.0

Change azure-identity to >=1.5.0

Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Ian Hellen <ianhelle@microsoft.com>

Some miscellaneous fixes (microsoft#115)

* Some miscellaneous fixes including one to mordor_driver (Mitre web site change)
Adding two scripts for testing pre-release msticpy.
Some additions to VTLookupV3.ipynb text and trapping for import of nest_asyncio without installing

* Updating version

* Adding azure-mgmt-core to requirements.txt

Adding exceptions to test_pkg_imports.py to skip vt and vt_graph_api (now extras)

* Updating .pre-commit-config.yaml to exclude tests from pylint and flake8 checks

* Adding code avoidance if AzureCLI section is not in msticpconfig.yaml

Updated notebook so that it has data.

* Skipping this test since it no longer fails because of previous fix that skips Env credentials if AzureCLI section not in configuration.

* Skipping this azure_data test