[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Monitoring #1893

Merged
merged 6 commits into from
Mar 3, 2020
Merged

Add Monitoring #1893

merged 6 commits into from
Mar 3, 2020

Conversation

alles-klar
Copy link
Contributor

Add the option to export django and nginx metrics for monitoring. It is disabled by default.

Prober documentation will follow but it should be more of less self-explaining.


Note: DefectDojo is now on Python3 and Django 2.2.1 Please submit your pull requests to the 'dev' branch as the 'legacy-python2.7' branch is only for bug fixes. Any new features submitted to the legacy branch will be ignored and closed.

When submitting a pull request, please make sure you have completed the following checklist:

  • Your code is flake8 compliant
  • Your code is python 3.5 compliant
  • If this is a new feature and not a bug fix, you've included the proper documentation in the ReadTheDocs documentation folder. https://github.com/DefectDojo/Documentation/tree/master/docs or provide feature documentation in the PR.
  • Model changes must include the necessary migrations in the dojo/dd_migrations folder.
  • Add applicable tests to the unit tests.

Copy link
Member
@valentijnscholten valentijnscholten left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it

@madchap
Copy link
Contributor
madchap commented Feb 21, 2020

So do I :)

@alles-klar any screenshots or otherwise data you can show us to demonstrate all it gathers?

Would need to give it a shot too.

@alles-klar
Copy link
Contributor Author

I haven't build a grafana dashboard jet and the django dashboard from the grafana community doesn't seem to work with the current version. But the following data is provided by default:

nginx status:

Active connections: 3 
server accepts handled requests
 115 115 503 
Reading: 0 Writing: 1 Waiting: 2

django metrics:

# HELP python_gc_objects_collected_total Objects collected during gc
# TYPE python_gc_objects_collected_total counter
python_gc_objects_collected_total{generation="0"} 11738.0
python_gc_objects_collected_total{generation="1"} 3217.0
python_gc_objects_collected_total{generation="2"} 321.0
# HELP python_gc_objects_uncollectable_total Uncollectable object found during GC
# TYPE python_gc_objects_uncollectable_total counter
python_gc_objects_uncollectable_total{generation="0"} 0.0
python_gc_objects_uncollectable_total{generation="1"} 0.0
python_gc_objects_uncollectable_total{generation="2"} 0.0
# HELP python_gc_collections_total Number of times this generation was collected
# TYPE python_gc_collections_total counter
python_gc_collections_total{generation="0"} 550.0
python_gc_collections_total{generation="1"} 50.0
python_gc_collections_total{generation="2"} 4.0
# HELP python_info Python platform information
# TYPE python_info gauge
python_info{implementation="CPython",major="3",minor="7",patchlevel="6",version="3.7.6"} 1.0
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.026404352e+09
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 1.39030528e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.5825621324e+09
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 3.16
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 8.0
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP django_http_requests_before_middlewares_total Total count of requests before middlewares run.
# TYPE django_http_requests_before_middlewares_total counter
django_http_requests_before_middlewares_total 30.0
# TYPE django_http_requests_before_middlewares_created gauge
django_http_requests_before_middlewares_created 1.5825621337711346e+09
# HELP django_http_responses_before_middlewares_total Total count of responses before middlewares run.
# TYPE django_http_responses_before_middlewares_total counter
django_http_responses_before_middlewares_total 29.0
# TYPE django_http_responses_before_middlewares_created gauge
django_http_responses_before_middlewares_created 1.5825621337711704e+09
# HELP django_http_requests_latency_including_middlewares_seconds Histogram of requests processing time (including middleware processing time).
# TYPE django_http_requests_latency_including_middlewares_seconds histogram
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.005"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.01"} 0.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.025"} 23.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.05"} 27.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.075"} 27.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.1"} 28.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.25"} 28.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.5"} 28.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="0.75"} 28.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="1.0"} 28.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="2.5"} 29.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="5.0"} 29.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="7.5"} 29.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="10.0"} 29.0
django_http_requests_latency_including_middlewares_seconds_bucket{le="+Inf"} 29.0
django_http_requests_latency_including_middlewares_seconds_count 29.0
django_http_requests_latency_including_middlewares_seconds_sum 1.710299098995165
# TYPE django_http_requests_latency_including_middlewares_seconds_created gauge
django_http_requests_latency_including_middlewares_seconds_created 1.5825621337712102e+09
# HELP django_http_requests_unknown_latency_including_middlewares_total Count of requests for which the latency was unknown (when computing django_http_requests_latency_including_middlewares_seconds).
# TYPE django_http_requests_unknown_latency_including_middlewares_total counter
django_http_requests_unknown_latency_including_middlewares_total 0.0
# TYPE django_http_requests_unknown_latency_including_middlewares_created gauge
django_http_requests_unknown_latency_including_middlewares_created 1.5825621337713187e+09
# HELP django_http_requests_latency_seconds_by_view_method Histogram of request processing time labelled by view.
# TYPE django_http_requests_latency_seconds_by_view_method histogram
django_http_requests_latency_seconds_by_view_method_bucket{le="0.01",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.025",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.05",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.075",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.1",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.25",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.5",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.75",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="1.0",method="GET",view="home"} 10.0
django_http_requests_latency_seconds_by_view_method_bucket{le="2.5",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="5.0",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="7.5",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="10.0",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="25.0",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="50.0",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="75.0",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_bucket{le="+Inf",method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_count{method="GET",view="home"} 11.0
django_http_requests_latency_seconds_by_view_method_sum{method="GET",view="home"} 1.1733497370005352
django_http_requests_latency_seconds_by_view_method_bucket{le="0.01",method="GET",view="login"} 0.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.025",method="GET",view="login"} 15.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.05",method="GET",view="login"} 17.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.075",method="GET",view="login"} 17.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.1",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.25",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.5",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="0.75",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="1.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="2.5",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="5.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="7.5",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="10.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="25.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="50.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="75.0",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_bucket{le="+Inf",method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_count{method="GET",view="login"} 18.0
django_http_requests_latency_seconds_by_view_method_sum{method="GET",view="login"} 0.36931576600909466
# TYPE django_http_requests_latency_seconds_by_view_method_created gauge
django_http_requests_latency_seconds_by_view_method_created{method="GET",view="home"} 1.582562254519147e+09
django_http_requests_latency_seconds_by_view_method_created{method="GET",view="login"} 1.582562263442309e+09
# HELP django_http_requests_unknown_latency_total Count of requests for which the latency was unknown.
# TYPE django_http_requests_unknown_latency_total counter
django_http_requests_unknown_latency_total 0.0
# TYPE django_http_requests_unknown_latency_created gauge
django_http_requests_unknown_latency_created 1.5825621337714148e+09
# HELP django_http_ajax_requests_total Count of AJAX requests.
# TYPE django_http_ajax_requests_total counter
django_http_ajax_requests_total 0.0
# TYPE django_http_ajax_requests_created gauge
django_http_ajax_requests_created 1.5825621337714386e+09
# HELP django_http_requests_total_by_method_total Count of requests by method.
# TYPE django_http_requests_total_by_method_total counter
django_http_requests_total_by_method_total{method="GET"} 30.0
# TYPE django_http_requests_total_by_method_created gauge
django_http_requests_total_by_method_created{method="GET"} 1.5825622533540692e+09
# HELP django_http_requests_total_by_transport_total Count of requests by transport.
# TYPE django_http_requests_total_by_transport_total counter
django_http_requests_total_by_transport_total{transport="https"} 30.0
# TYPE django_http_requests_total_by_transport_created gauge
django_http_requests_total_by_transport_created{transport="https"} 1.5825622533541088e+09
# HELP django_http_requests_total_by_view_transport_method_total Count of requests by view, transport, method.
# TYPE django_http_requests_total_by_view_transport_method_total counter
django_http_requests_total_by_view_transport_method_total{method="GET",transport="https",view="home"} 11.0
django_http_requests_total_by_view_transport_method_total{method="GET",transport="https",view="login"} 18.0
django_http_requests_total_by_view_transport_method_total{method="GET",transport="https",view="prometheus-django-metrics"} 1.0
# TYPE django_http_requests_total_by_view_transport_method_created gauge
django_http_requests_total_by_view_transport_method_created{method="GET",transport="https",view="home"} 1.582562254358662e+09
django_http_requests_total_by_view_transport_method_created{method="GET",transport="https",view="login"} 1.582562263362379e+09
django_http_requests_total_by_view_transport_method_created{method="GET",transport="https",view="prometheus-django-metrics"} 1.582562498993328e+09
# HELP django_http_requests_body_total_bytes Histogram of requests by body size.
# TYPE django_http_requests_body_total_bytes histogram
django_http_requests_body_total_bytes_bucket{le="0.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="1.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="2.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="4.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="8.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="16.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="32.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="64.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="128.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="256.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="512.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="1024.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="2048.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="4096.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="8192.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="16384.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="32768.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="65536.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="131072.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="262144.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="524288.0"} 30.0
django_http_requests_body_total_bytes_bucket{le="1.048576e+06"} 30.0
django_http_requests_body_total_bytes_bucket{le="2.097152e+06"} 30.0
django_http_requests_body_total_bytes_bucket{le="4.194304e+06"} 30.0
django_http_requests_body_total_bytes_bucket{le="8.388608e+06"} 30.0
django_http_requests_body_total_bytes_bucket{le="1.6777216e+07"} 30.0
django_http_requests_body_total_bytes_bucket{le="3.3554432e+07"} 30.0
django_http_requests_body_total_bytes_bucket{le="6.7108864e+07"} 30.0
django_http_requests_body_total_bytes_bucket{le="1.34217728e+08"} 30.0
django_http_requests_body_total_bytes_bucket{le="2.68435456e+08"} 30.0
django_http_requests_body_total_bytes_bucket{le="5.36870912e+08"} 30.0
django_http_requests_body_total_bytes_bucket{le="+Inf"} 30.0
django_http_requests_body_total_bytes_count 30.0
django_http_requests_body_total_bytes_sum 0.0
# TYPE django_http_requests_body_total_bytes_created gauge
django_http_requests_body_total_bytes_created 1.5825621337715633e+09
# HELP django_http_responses_total_by_templatename_total Count of responses by template name.
# TYPE django_http_responses_total_by_templatename_total counter
django_http_responses_total_by_templatename_total{templatename="['dojo/login.html']"} 18.0
# TYPE django_http_responses_total_by_templatename_created gauge
django_http_responses_total_by_templatename_created{templatename="['dojo/login.html']"} 1.5825622633721159e+09
# HELP django_http_responses_total_by_status_total Count of responses by status.
# TYPE django_http_responses_total_by_status_total counter
django_http_responses_total_by_status_total{status="302"} 11.0
django_http_responses_total_by_status_total{status="200"} 18.0
# TYPE django_http_responses_total_by_status_created gauge
django_http_responses_total_by_status_created{status="302"} 1.5825622545189748e+09
django_http_responses_total_by_status_created{status="200"} 1.5825622634422226e+09
# HELP django_http_responses_total_by_status_view_method_total Count of responses by status, view, method.
# TYPE django_http_responses_total_by_status_view_method_total counter
django_http_responses_total_by_status_view_method_total{method="GET",status="302",view="home"} 11.0
django_http_responses_total_by_status_view_method_total{method="GET",status="200",view="login"} 18.0
# TYPE django_http_responses_total_by_status_view_method_created gauge
django_http_responses_total_by_status_view_method_created{method="GET",status="302",view="home"} 1.582562254519003e+09
django_http_responses_total_by_status_view_method_created{method="GET",status="200",view="login"} 1.5825622634422457e+09
# HELP django_http_responses_body_total_bytes Histogram of responses by body size.
# TYPE django_http_responses_body_total_bytes histogram
django_http_responses_body_total_bytes_bucket{le="0.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="1.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="2.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="4.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="8.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="16.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="32.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="64.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="128.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="256.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="512.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="1024.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="2048.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="4096.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="8192.0"} 11.0
django_http_responses_body_total_bytes_bucket{le="16384.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="32768.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="65536.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="131072.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="262144.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="524288.0"} 29.0
django_http_responses_body_total_bytes_bucket{le="1.048576e+06"} 29.0
django_http_responses_body_total_bytes_bucket{le="2.097152e+06"} 29.0
django_http_responses_body_total_bytes_bucket{le="4.194304e+06"} 29.0
django_http_responses_body_total_bytes_bucket{le="8.388608e+06"} 29.0
django_http_responses_body_total_bytes_bucket{le="1.6777216e+07"} 29.0
django_http_responses_body_total_bytes_bucket{le="3.3554432e+07"} 29.0
django_http_responses_body_total_bytes_bucket{le="6.7108864e+07"} 29.0
django_http_responses_body_total_bytes_bucket{le="1.34217728e+08"} 29.0
django_http_responses_body_total_bytes_bucket{le="2.68435456e+08"} 29.0
django_http_responses_body_total_bytes_bucket{le="5.36870912e+08"} 29.0
django_http_responses_body_total_bytes_bucket{le="+Inf"} 29.0
django_http_responses_body_total_bytes_count 29.0
django_http_responses_body_total_bytes_sum 185220.0
# TYPE django_http_responses_body_total_bytes_created gauge
django_http_responses_body_total_bytes_created 1.5825621337719688e+09
# HELP django_http_responses_total_by_charset_total Count of responses by charset.
# TYPE django_http_responses_total_by_charset_total counter
django_http_responses_total_by_charset_total{charset="utf-8"} 29.0
# TYPE django_http_responses_total_by_charset_created gauge
django_http_responses_total_by_charset_created{charset="utf-8"} 1.582562254519058e+09
# HELP django_http_responses_streaming_total Count of streaming responses.
# TYPE django_http_responses_streaming_total counter
django_http_responses_streaming_total 0.0
# TYPE django_http_responses_streaming_created gauge
django_http_responses_streaming_created 1.582562133772206e+09
# HELP django_http_exceptions_total_by_type_total Count of exceptions by object type.
# TYPE django_http_exceptions_total_by_type_total counter
# HELP django_http_exceptions_total_by_view_total Count of exceptions by view.
# TYPE django_http_exceptions_total_by_view_total counter
# HELP django_model_inserts_total Number of insert operations by model.
# TYPE django_model_inserts_total counter
# HELP django_model_updates_total Number of update operations by model.
# TYPE django_model_updates_total counter
# HELP django_model_deletes_total Number of delete operations by model.
# TYPE django_model_deletes_total counter
# HELP django_migrations_unapplied_total Count of unapplied migrations by database connection
# TYPE django_migrations_unapplied_total gauge
# HELP django_migrations_applied_total Count of applied migrations by database connection
# TYPE django_migrations_applied_total gauge
# HELP django_db_new_connections_total Counter of created connections by database and by vendor.
# TYPE django_db_new_connections_total counter
django_db_new_connections_total{alias="default",vendor="postgresql"} 29.0
# TYPE django_db_new_connections_created gauge
django_db_new_connections_created{alias="default",vendor="postgresql"} 1.5825622537070813e+09
# HELP django_db_new_connection_errors_total Counter of connection failures by database and by vendor.
# TYPE django_db_new_connection_errors_total counter
# HELP django_db_execute_total Counter of executed statements by database and by vendor, including bulk executions.
# TYPE django_db_execute_total counter
django_db_execute_total{alias="default",vendor="postgresql"} 87.0
# TYPE django_db_execute_created gauge
django_db_execute_created{alias="default",vendor="postgresql"} 1.5825622537129788e+09
# HELP django_db_execute_many_total Counter of executed statements in bulk operations by database and by vendor.
# TYPE django_db_execute_many_total counter
# HELP django_db_errors_total Counter of execution errors by database, vendor and exception type.
# TYPE django_db_errors_total counter

@alles-klar
Copy link
Contributor Author

@valentijnscholten everything ok for you? If yes please approve the PR, so we can merge it.

@valentijnscholten
Copy link
Member

Sorry, I thought I already had. Now that I am looking at it:

  • The postgres change, I have no idea about the impact
  • If I understand it correcty by default there is no authentication on metrics? That would be ok, if it is off by default. But it looks enabled for nginx by default? My first instinct would be to have it off by default for production/release mode.

@alles-klar
Copy link
Contributor Author

Sorry, I thought I already had. Now that I am looking at it:

  • The postgres change, I have no idea about the impact

Should not have any impact: https://stackoverflow.com/questions/47946856/what-is-the-difference-between-postgres-and-postgresql-psycopg2-as-a-database-en/47946916

  • If I understand it correcty by default there is no authentication on metrics? That would be ok, if it is off by default. But it looks enabled for nginx by default? My first instinct would be to have it off by default for production/release mode.

You are right. I have to figure out if we can easily turn off the nginx metrics by default. On the other hand the nginx metrics doesn't provide much information.

Dockerfile.nginx Outdated Show resolved Hide resolved
@alles-klar
Copy link
Contributor Author

From my perspective it should be fine now.

@alles-klar alles-klar merged commit 0c41cec into DefectDojo:dev Mar 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants