tag:status.circonus.com,2005:/historyCirconus Status - Incident History2024-03-29T05:22:41ZCirconustag:status.circonus.com,2005:Incident/191103132023-12-19T21:43:35Z2023-12-19T21:43:35ZPublic Broker Fleet refresh<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>21:43</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>16:16</var> UTC</small><br><strong>Update</strong> - Customers are encouraged to double-check any IP whitelisting before checks are migrated on, or after, Dec 19.</p><p><small>Nov <var data-var='date'>13</var>, <var data-var='time'>19:06</var> UTC</small><br><strong>Scheduled</strong> - On 2023-12-19 Circonus will be retiring the following brokers and replacing them with updated nodes in the same geolocations. Customers are encouraged to move existing checks and agents to the new brokers, and update any firewall rules (access lists/whitelists) they may have in advance of this date. On, or shortly after, 2023-12-19 any remaining checks will be migrated automatically to the new corresponding broker.<br /><br />Contact support@circonus.com for any questions.<br /><br />Amsterdam (nl1-3) is going to "Circonus Europe West 1" 188.166.135.74<br />Ashburn (us1-4) is going to "Circonus US East 1" 35.199.0.16<br />Chicago (us3-2) is going to "Circonus US Central 1" 35.184.145.252<br />London (uk1-1) is going to "Circonus Europe West 2" 206.189.245.166<br />Munich (de1-3) is going to "Circonus Europe Central 1" 174.138.103.181<br />Nagano (jp1-2) is going to "Circonus Asia Northeast 1" 34.84.94.26<br />San Jose (us2-4) is going to "Circonus US West 1" 24.199.69.248<br />Singapore (sg1-1) is going to "Circonus Asia Southeast 1" 139.59.194.220</p>tag:status.circonus.com,2005:Incident/194239862023-12-13T12:56:15Z2023-12-13T12:56:15ZMunich broker is currently down<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>12:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>12:55</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>04:08</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>04:04</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>04:00</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/194225842023-12-13T01:56:37Z2023-12-13T01:56:37ZThe Munich Publicbroker is currently down.<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>01:56</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>23:56</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>23:47</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/188174272023-10-16T20:12:31Z2023-10-16T20:12:31ZPassport Down<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>20:12</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>19:11</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>19:06</var> UTC</small><br><strong>Investigating</strong> - Our IdP vendor is experiencing a service outage which is preventing users from logging into Passport.</p>tag:status.circonus.com,2005:Incident/184485832023-09-09T18:08:02Z2023-09-09T18:08:02ZIssue with login page<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>18:08</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>17:44</var> UTC</small><br><strong>Monitoring</strong> - We believe we have resolved the issue. We will continue to monitor.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>17:19</var> UTC</small><br><strong>Update</strong> - Login has been restored for non-SAML logins. SAML logins are still being addressed. Existing sessions are not affected.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>16:00</var> UTC</small><br><strong>Investigating</strong> - We are currently experiencing an issue with the login page. We are investigating, and we will provide an update shortly.</p>tag:status.circonus.com,2005:Incident/181092302023-08-10T20:45:29Z2023-08-10T20:45:29ZIngestion delay on public metric cluster<p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>10</var>, <var data-var='time'>20:18</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/179809022023-07-28T00:57:13Z2023-07-28T00:57:13ZAshburn broker network partition<p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>00:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>00:45</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jul <var data-var='date'>28</var>, <var data-var='time'>00:18</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/176942232023-06-27T01:40:23Z2023-06-27T01:42:53ZPrimary SaaS CAQLBroker metrics using fill:forward() not working<p><small>Jun <var data-var='date'>27</var>, <var data-var='time'>01:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>26</var>, <var data-var='time'>17:30</var> UTC</small><br><strong>Investigating</strong> - CAQL Broker metrics on the main SaaS account that used the `fill:forward()` function were no longer emitting values. Private CAQLBrokers, and CAQL-based graphs were not affected.</p>tag:status.circonus.com,2005:Incident/164019192023-03-10T02:26:05Z2023-03-10T02:26:05ZGraph data inconsistently loading<p><small>Mar <var data-var='date'>10</var>, <var data-var='time'>02:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 9</var>, <var data-var='time'>14:28</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an issue where certain graphs in our SaaS environment are not fully loading. The temporary work around for this issue is to refresh the page, potentially a few times, until the data appears.</p>tag:status.circonus.com,2005:Incident/161284272023-02-23T02:20:08Z2023-02-23T02:20:08ZUI/API Feb 22nd 6pm Pacific / 9pm Eastern / Feb23 02 UTC<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>02:20</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>02:18</var> UTC</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>02:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>13</var>, <var data-var='time'>21:51</var> UTC</small><br><strong>Scheduled</strong> - Circonus will be upgrading a core architectural component. During this time, the Web UI, and API will be unavailable. Metric ingestion, aggregation, alerting, and notification will continue to operate during this time. Grafana graphs, and alerting through Grafana will also continue to operate during this time.</p>tag:status.circonus.com,2005:Incident/159794502023-01-31T05:53:03Z2023-01-31T05:53:03ZIngestion delay<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>05:53</var> UTC</small><br><strong>Resolved</strong> - All services have returned to normal.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>00:03</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating ingestion delay for some customers.</p>tag:status.circonus.com,2005:Incident/158940022023-01-20T23:32:10Z2023-01-20T23:32:10ZGraphs behind due to metric ingestion delay for some customers<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>17:58</var> UTC</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.circonus.com,2005:Incident/128828532022-11-12T03:06:30Z2022-11-12T03:06:30ZMetric ingestion delay on some metrics using public trap brokers<p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>03:06</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>03:06</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>02:45</var> UTC</small><br><strong>Monitoring</strong> - The broker has been fixed, we on monitoring as it finishes catching up.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>01:17</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/112691532022-09-26T09:54:00Z2022-09-26T09:59:08ZDelay in alert notifications for some customers<p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>09:54</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>09:53</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'>26</var>, <var data-var='time'>09:10</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/110371622022-09-01T23:48:16Z2022-09-01T23:48:16ZAlerting delays<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>23:48</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>23:47</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>23:34</var> UTC</small><br><strong>Investigating</strong> - Alerting may be delayed for some customers. We are investigating</p>tag:status.circonus.com,2005:Incident/110305642022-09-01T19:54:48Z2022-09-01T19:54:48ZNagano broker down<p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>19:54</var> UTC</small><br><strong>Resolved</strong> - This broker is returned to service. Its IP address has changed to 34.146.69.2 and is located closer to Tokyo now, rather than the original Nagano.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>18:11</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>09:05</var> UTC</small><br><strong>Update</strong> - Additional provider environment attempts have been unsuccessful. We will be leaving this broker down for an extended period.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>08:03</var> UTC</small><br><strong>Identified</strong> - The new provider environment does not appear compatible with our broker checks cause incorrect results to be returned. We have shut this broker back down.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>07:34</var> UTC</small><br><strong>Monitoring</strong> - This broker has been moved to a new hosting provider. We were unable to keep the IP as part of this, and the broker is currently up on `34.84.180.175` for customers that whitelist checks from the broker. The broker is up, but we are investigating if all checks are running correctly or not.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>06:27</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Sep <var data-var='date'> 1</var>, <var data-var='time'>05:27</var> UTC</small><br><strong>Investigating</strong> - Due to vendor, Nagano broker is down, and could be down for an extended period</p>tag:status.circonus.com,2005:Incident/108636842022-08-19T23:32:42Z2022-08-20T00:03:50ZCirconus Alerts Service Outage<p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>23:32</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>23:31</var> UTC</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>22:27</var> UTC</small><br><strong>Investigating</strong> - We are investigating an issue impacting Circonus Alerts. The Circonus Alerts service is experiencing an outage.</p>tag:status.circonus.com,2005:Incident/108150042022-08-12T19:35:43Z2022-08-12T19:35:43ZTrap broker delay<p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>19:35</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>12</var>, <var data-var='time'>19:16</var> UTC</small><br><strong>Identified</strong> - Customers using our public http-trap brokers may have some delay in ingestion.</p>tag:status.circonus.com,2005:Incident/107194322022-07-29T20:13:19Z2022-07-29T20:18:51ZSSO login issues and API check updates<p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>20:13</var> UTC</small><br><strong>Resolved</strong> - This issue is resolved. API calls for updating checks were also affected during this time.</p><p><small>Jul <var data-var='date'>29</var>, <var data-var='time'>19:33</var> UTC</small><br><strong>Investigating</strong> - We're seeing issues with all SSO providers for login.circonus.com, and are currently investigating it.</p>tag:status.circonus.com,2005:Incident/106925022022-07-26T10:17:17Z2022-07-26T10:17:17ZNagano broker down<p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>10:17</var> UTC</small><br><strong>Resolved</strong> - The hosting maintenance is over, and nagano1-jp.broker.circonus.net is up again.</p><p><small>Jul <var data-var='date'>26</var>, <var data-var='time'>10:06</var> UTC</small><br><strong>Identified</strong> - nagano1-jp.broker.circonus.net is down for unplanned host maintenance.</p>tag:status.circonus.com,2005:Incident/104743652022-07-05T20:45:51Z2022-07-05T20:45:51ZSSO logins not working<p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Resolved</strong> - This has been resolved.</p><p><small>Jul <var data-var='date'> 5</var>, <var data-var='time'>20:39</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue with new logins using common SSO providers (Google, Github, Slack, etc.) and are currently investigating.</p>tag:status.circonus.com,2005:Incident/103892372022-06-22T22:36:00Z2022-06-23T17:55:34ZIngestion delayed<p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>22:36</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>22</var>, <var data-var='time'>22:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.circonus.com,2005:Incident/98774592022-04-29T08:26:54Z2022-04-29T08:26:54ZMetric storage delay for some customers<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>08:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>07:20</var> UTC</small><br><strong>Update</strong> - http trap brokers are still behind on storage feeds, but are catching up. Alerting had no issue.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>06:59</var> UTC</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>06:59</var> UTC</small><br><strong>Investigating</strong> - Circonus presently has delays on storing metrics, but all alerting feeds have no delays. Storage is currently catching up.</p>tag:status.circonus.com,2005:Incident/96831682022-04-01T07:06:03Z2022-04-01T16:31:27ZAPI intermittently unavailable<p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>07:06</var> UTC</small><br><strong>Resolved</strong> - We've addressed the cause, and the API is responding normally to all requests again.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>06:43</var> UTC</small><br><strong>Investigating</strong> - api.circonus.com is presently throwing errors for some requests.</p>tag:status.circonus.com,2005:Incident/94756352022-03-06T21:00:00Z2022-03-06T21:15:11ZDNS outage<p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>21:00</var> UTC</small><br><strong>Resolved</strong> - Circonus' DNS provider partially stopped resolving DNS queries from 20:50 UTC to 21:00 UTC, potentially causing erratic web ui and api behavior. Service is now fully restored.</p>