Logs
The logs should be your first stop when troubleshooting any Galaxy issue. When working with Galaxy support, an agent will usually begin by consulting your logs. If a clearly noted issue has been marked there, you can save time by reading the logs and iterating on your code until the error goes away.
You can reach the logs dashboard by clicking on your app, then clicking the "Logs" tab.
There are several logging filters you can click, below the "Logs" tab itself.
The main options are to see logs in Real-time or filter By Date:
- Real-time: this allows you to watch your logs in real-time. You can click on View older to see older logs, click on the bottom to View new logs, and set the logs to Autoscrolling.
- By Date: this allows you to filter your logs by date and time. Even when you filter by date, we show logs paginated. You can still click on View older in case the logs you want to see are not shown.
These are the options to filter logs by type:
- All: for the combined view.
- App: for application-specific messages and errors.
- Service: for messages generated during the build process, and messages saying that a container is starting or stopping.
- Errors: exclusively for errors, which are flagged in red in other tab views.
Breakdown by tab will be especially useful if you have multiple containers running, and are regularly deploying.
There is also a button where you can download all the logs that are still available (we retain logs for 7 days).
Every message and error will be shown here.
If your containers are working correctly and have been up for a while, this may be the most convenient tab to check. If you have multiple containers and builds succeeding and failing, this view may be too crowded.
Any messages or errors thrown by your app will be shown here, assuming our image builder was able to build your app and successfully launch it.
Check this to see what your app is doing during normal operations. You can always add lines to your code to print what your app is doing for added visibility.
If your app isn't successfully deploying, this is the tab you'll want to check. Our image builder tries to build your app into a container; if it fails, the failure will be noted here.
One type of error (example here) is that a particular version of a package wasn't able to be built by our image builder. If this happens to your app, the output will frequently include a line like this, after the package install line:
node-pre-gyp install --fallback-to-build
A working solution in such cases is usually to revert to an earlier version of the package.
Errors will be shown in this tab. Check to see if there are any outstanding errors if your app is malfunctioning.
If any errors or messages thrown by your app are older than 1 week, they won't appear in your logs. For more permanent log storage options, contact support@meteor.com.
Sometimes, you may prefer to download your logs to search through them or send the logs to someone without access to your Galaxy app. You can download your logs on the right side of the filter options.
The default value is set to 1 day. You can select from 1 to 7 days. Galaxy will export your logs and send you by email as soon as they are ready.
If you would like access to older logs and more flexibility in how you access your logs, you can run your own Elasticsearch server and configure Galaxy to send your app's logs to your own server. You can then use any tools that support Elasticsearch to search and process your logs. Several companies provide hosted Elasticsearch systems, including Elastic Cloud and Amazon Web Services.
To use Galaxy's custom log storage support, set up your Elasticsearch server, and provide its URL as the USER_LOG_DESTINATION environment variables in your app's settings.json. Galaxy will send your app's standard output and standard error, as well as notifications of container start and exit events, to your Elasticsearch server. Note that logs from building containers, as well as some other service logs from the Galaxy scheduler, are not at this time sent to your Elasticsearch server.
You will still be able to view your last week of logs in the Galaxy dashboard.
Logs are written to Elasticsearch indices with names that look like app_logs-2018-01-29. A new index is created for each day (in the UTC time zone). This lets you reclaim space on your server easily by deleting old indices.
We recommend you create an index template to define the fields used by Galaxy's Elasticsearch integration. The index template tells Elasticsearch that the @timestamp field contains a date and time, and that several other fields which contain IDs such as container ID not be "analyzed": ie, they should be treated literally instead of broken into strings and lower-cased.
Elasticsearch has changed some of its APIs in recent versions, so the exact index template depends on your version of Elasticsearch.
For versions of Elasticsearch earlier than v6, run the following code at your shell (substituting in your Elasticsearch server's credentials and address) to create the index template:
For Elasticsearch v6, run the following code at your shell (substituting in your Elasticsearch server's credentials and address) to create the index template:
For Elasticsearch v7, run the following command to create the index pattern:
You can reach the logs dashboard by clicking on your app, then clicking the "Logs" tab.
Log Filters
There are several logging filters you can click, below the "Logs" tab itself.
The main options are to see logs in Real-time or filter By Date:
- Real-time: this allows you to watch your logs in real-time. You can click on View older to see older logs, click on the bottom to View new logs, and set the logs to Autoscrolling.
- By Date: this allows you to filter your logs by date and time. Even when you filter by date, we show logs paginated. You can still click on View older in case the logs you want to see are not shown.
These are the options to filter logs by type:
- All: for the combined view.
- App: for application-specific messages and errors.
- Service: for messages generated during the build process, and messages saying that a container is starting or stopping.
- Errors: exclusively for errors, which are flagged in red in other tab views.
Breakdown by tab will be especially useful if you have multiple containers running, and are regularly deploying.
There is also a button where you can download all the logs that are still available (we retain logs for 7 days).
All
Every message and error will be shown here.
If your containers are working correctly and have been up for a while, this may be the most convenient tab to check. If you have multiple containers and builds succeeding and failing, this view may be too crowded.
App
Any messages or errors thrown by your app will be shown here, assuming our image builder was able to build your app and successfully launch it.
Check this to see what your app is doing during normal operations. You can always add lines to your code to print what your app is doing for added visibility.
Service
If your app isn't successfully deploying, this is the tab you'll want to check. Our image builder tries to build your app into a container; if it fails, the failure will be noted here.
One type of error (example here) is that a particular version of a package wasn't able to be built by our image builder. If this happens to your app, the output will frequently include a line like this, after the package install line:
node-pre-gyp install --fallback-to-build
A working solution in such cases is usually to revert to an earlier version of the package.
Errors
Errors will be shown in this tab. Check to see if there are any outstanding errors if your app is malfunctioning.
Older Logs
If any errors or messages thrown by your app are older than 1 week, they won't appear in your logs. For more permanent log storage options, contact support@meteor.com.
Download Logs
Sometimes, you may prefer to download your logs to search through them or send the logs to someone without access to your Galaxy app. You can download your logs on the right side of the filter options.
The default value is set to 1 day. You can select from 1 to 7 days. Galaxy will export your logs and send you by email as soon as they are ready.
Custom Log Storage
If you would like access to older logs and more flexibility in how you access your logs, you can run your own Elasticsearch server and configure Galaxy to send your app's logs to your own server. You can then use any tools that support Elasticsearch to search and process your logs. Several companies provide hosted Elasticsearch systems, including Elastic Cloud and Amazon Web Services.
To use Galaxy's custom log storage support, set up your Elasticsearch server, and provide its URL as the USER_LOG_DESTINATION environment variables in your app's settings.json. Galaxy will send your app's standard output and standard error, as well as notifications of container start and exit events, to your Elasticsearch server. Note that logs from building containers, as well as some other service logs from the Galaxy scheduler, are not at this time sent to your Elasticsearch server.
You will still be able to view your last week of logs in the Galaxy dashboard.
Logs are written to Elasticsearch indices with names that look like app_logs-2018-01-29. A new index is created for each day (in the UTC time zone). This lets you reclaim space on your server easily by deleting old indices.
We recommend you create an index template to define the fields used by Galaxy's Elasticsearch integration. The index template tells Elasticsearch that the @timestamp field contains a date and time, and that several other fields which contain IDs such as container ID not be "analyzed": ie, they should be treated literally instead of broken into strings and lower-cased.
Elasticsearch has changed some of its APIs in recent versions, so the exact index template depends on your version of Elasticsearch.
Setting up index templates for Elasticsearch 5 and earlier
For versions of Elasticsearch earlier than v6, run the following code at your shell (substituting in your Elasticsearch server's credentials and address) to create the index template:
curl -X PUT 'https://username:password@your-elasticsearch-server.com/_template/app_logs?pretty' -H 'Content-Type: application/json' -d '
{
"template": "app_logs-*",
"mappings": {
"line": {
"properties": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"appId": {
"type": "string",
"index": "not_analyzed"
},
"containerId": {
"type": "string",
"index": "not_analyzed"
},
"stack": {
"type": "string",
"index": "not_analyzed"
},
"log": {
"type": "string"
},
"stream": {
"type": "string",
"index": "not_analyzed"
},
"rootUrl": {
"type": "string",
"index": "not_analyzed"
},
"appVersionId": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
'Setting up index templates for Elasticsearch 6
For Elasticsearch v6, run the following code at your shell (substituting in your Elasticsearch server's credentials and address) to create the index template:
curl -X PUT 'https://username:password@your-elasticsearch-server.com/_template/app_logs?pretty' -H 'Content-Type: application/json' -d '
{
"index_patterns": ["app_logs-*"],
"mappings": {
"line": {
"properties": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"appId": {
"type": "keyword",
"index": true
},
"containerId": {
"type": "keyword",
"index": true
},
"stack": {
"type": "keyword",
"index": true
},
"log": {
"type": "text",
"index": true
},
"stream": {
"type": "keyword",
"index": true
},
"rootUrl": {
"type": "keyword",
"index": true
},
"appVersionId": {
"type": "keyword",
"index": true
}
}
}
}
}
'Setting up index templates for Elasticsearch 7 and later
For Elasticsearch v7, run the following command to create the index pattern:
curl -X PUT 'https://username:password@your-elasticsearch-server.com/_template/app_logs?pretty' -H 'Content-Type: application/json' -d '
{
"index_patterns": ["app_logs-*"],
"template": {
"mappings": {
"line": {
"properties": {
"@timestamp": {
"type": "date"
},
"appId": {
"type": "text"
},
"containerId": {
"type": "text"
},
"stack": {
"type": "text"
},
"log": {
"type": "text"
},
"stream": {
"type": "text"
},
"rootUrl": {
"type": "text"
},
"appVersionId": {
"type": "text"
}
}
}
}
}
}
'Updated on: 15/07/2024
Thank you!