We are continuing to monitor for any further issues.
Posted Apr 12, 2021 - 13:16 MDT
Update
The intermittent system issues have been resolved and all applications are now fully operational. We will continue to monitor our applications throughout the night to ensure these issues do not return.
Thank you for your patience.
-The Swipeclock Team
Posted Apr 08, 2021 - 21:22 MDT
Update
We are monitoring intermittent system access errors (gateway errors) being reported by users. Our team is working diligently to identify and resolve this issue. We will post information as it becomes available.
Thank you for your patience, -The Swipeclock Team
Posted Apr 08, 2021 - 19:02 MDT
Update
We continue to work toward full system restoration following our service disruption. While most system functionality has been restored, we have several key features and processes that are still being worked on. One of those is our Daily Auto Email. We are working around the clock to get these remaining features in place and will continue to update you here as progress is made.
Again, we appreciate your patience and understanding.
Thank you. -The Swipeclock Team
Posted Apr 07, 2021 - 08:51 MDT
Update
Current Status: All Swipeclock systems are back online as of early this morning. The final batches of time punches from hardware clocks are currently processing and should be complete within the next hour.
We are aware that the system has had periodic slowness as it came online. The performance has continued to improve throughout the day and will continue to see progress. We have appreciated everyone’s patience as performance scales.
We have resolved some issues that arose as the system came back online. At this point, please use the normal process of submitting a ticket for any issues you now experience.
All Timekeeping Apps are Functional: TimeWorksPlus TimeWorks Mobile TimeWorks Employee Mobile TimeWorks Plus API Webclock WorkforceHub API WorkforceHub TimeWorks
ApplicantStack Recruit, ApplicantStack Onboard, and TimeSimplicity were unaffected.
While the independent forensic team is still investigating, they have made significant progress and there continues to be no evidence of theft of data from our servers.
Next Steps: Thank you for your patience on the usage of the system today and for allowing those with critical payrolls to process. We will continue to provide updates on our Status Page online: http://status.payrollservers.us/
Resources: We have prepared a Recovery Guide that provides additional details and instructions. Please refer to this guide below.
Thank you,
—Swipeclock Team
RECOVERY GUIDE Late on April 1st, we detected suspicious activity within our cloud environment which impacted the availability to access our timekeeping software.
This service interruption also prevented employees from punching in via WebClock and mobile. Hardware clocks were still able to collect and store punches but could not transmit punch data to the system.
Our development team restored limited service on April 3rd which, in most instances, allowed employees to resume using WebClock and mobile. This limited service also enabled hardware clocks to transmit punch data, collected during the outage, to our servers.
Full access to our timekeeping applications for both managers and employees is now available.
Once you and your clients have access to the system, please keep in mind the following:
Time card entries for employees that were unable to punch in via WebClock and mobile will need to be manually entered. Because this outage affected the software side of our timekeeping system, WebClock, TimeWorksPlus Mobile, and TimeWorksEmployee Mobile could not collect punches. This disruption started at approximately 11:50 PM MT on April 1st and lasted until approximately 6:00 PM MT April 3rd. However, the time at which you and your client see punches resume will depend on when employees started using the clock after the outage. Punches made on physical clocks will be processed and populate the time card, but you may not see them immediately. As mentioned earlier, physical clocks were still able to collect punches during the outage. Once limited service was restored on April 3rd, those clocks were able to transmit that data to our servers.
Please note that if your client is using a basic hardware clock, like Punch+, they may need to press “9” on the keypad to force the transmission to our servers.
Processing this backlog of punches from the servers to the time card is ongoing, and at this time, we are unsure how long it will take. If you don’t see all punch data when you first log into the system, it may still be coming. We will keep our providers updated with this progress through status updates.
All other timekeeping data, like time card edits, accrual earnings, and time-off requests, should be restored as it was before the outage. Our database is in the process of being restored and all data prior to the outage is expected to be recovered.
Please ask your clients to log into the system later today to review these items to see if there are any errors or problems.
As mentioned, data will take a while to populate. Wait a while before checking for data. If you experience any additional issues, please send a support ticket to support@swipeclock.com
Posted Apr 05, 2021 - 16:18 MDT
Update
We are continuing to monitor for any further issues.
Posted Apr 05, 2021 - 14:28 MDT
Monitoring
Current Status: As mentioned on our update earlier this morning, all data - up until the time the system was taken down on the evening of April 1, 2021 - has been restored. This now includes all TimeWorks and TimeWorksPlus information with the exception of hardware clock punches. These hardware punches are currently being loaded into the system and will complete later today. (See Recovery Guide Below)
In addition, the full production system is online, providing for complete access for all timekeeping applications. However, if you do not need to run payroll or need immediate system access, please refrain from logging in to reduce the impact on the system to allow for continued testing through this afternoon.
All Timekeeping Apps are Functional: TimeWorksPlus TimeWorks Mobile TimeWorks Employee Mobile TimeWorks Plus API Webclock WorkforceHub API WorkforceHub TimeWorks
ApplicantStack Recruit, ApplicantStack Onboard, and TimeSimplicity were unaffected.
While the independent forensic team is still investigating, they have made significant progress and there continues to be no evidence of theft of data from our servers.
Next Steps: As a reminder, while the system is coming online, you may experience intermittent performance inconsistencies. We have prepared a Recovery Guide that provides additional details and instructions. Please refer to this guide below.
Thank you,
—Swipeclock Team
RECOVERY GUIDE Late on April 1st, we detected suspicious activity within our cloud environment which impacted the availability to access our timekeeping software. This service interruption also prevented employees from punching in via WebClock and mobile. Hardware clocks were still able to collect and store punches but could not transmit punch data to the system.
Our development team restored limited service on April 3rd which, in most instances, allowed employees to resume using WebClock and mobile. This limited service also enabled hardware clocks to transmit punch data, collected during the outage, to our servers.
Full access to our timekeeping applications for both managers and employees is now available.
Once you and your clients have access to the system, please keep in mind the following:
Time card entries for employees that were unable to punch in via WebClock and mobile will need to be manually entered. Because this outage affected the software side of our timekeeping system, WebClock, TimeWorksPlus Mobile, and TimeWorksEmployee Mobile could not collect punches. This disruption started at approximately 11:50 PM MT on April 1st and lasted until approximately 6:00 PM MT April 3rd. However, the time at which you and your client see punches resume will depend on when employees started using the clock after the outage.
Punches made on physical clocks will be processed and populate the time card, but you may not see them immediately. As mentioned earlier, physical clocks were still able to collect punches during the outage. Once limited service was restored on April 3rd, those clocks were able to transmit that data to our servers. Please note that if your client is using a basic hardware clock, like Punch+, they may need to press “9” on the keypad to force the transmission to our servers. Processing this backlog of punches from the servers to the time card is ongoing, and at this time, we are unsure how long it will take. If you don’t see all punch data when you first log into the system, it may still be coming. We will keep our providers updated with this progress through status updates.
All other timekeeping data, like time card edits, accrual earnings, and time off requests, should be restored as it was before the outage. Our database is in the process of being restored and all data prior to the outage is expected to be recovered.
Please ask your clients to log into the system later today to review these items to see if there are any errors or problems.
As mentioned, data will take a while to populate. Wait a while before checking for data. If you experience any additional issues, please send a support ticket to support@swipeclock.com
Posted Apr 05, 2021 - 08:10 MDT
Update
Current Status: We have now restored all data up until the time the system was taken down on April 1 with the exception of data in TimeWorks sites. (legacy system) This data is being added now but we ask that you NOT MAKE ANY EDITS to TimeWorks site data until a full recovery has occurred.
Next Steps: As we test the system for bringing it back online, it is possible for users to log in to the timekeeping management console. We ask that partners and clients alike refrain from doing so until all testing is complete and we provide all clear here and in an official email update. It is likely the system may see service interruptions as we continue to test and we don't want users to experience any issues.
Timeline: We will continue to provide frequent updates on our Status Page until the system is fully online: http://status.payrollservers.us/
We want to thank our customers, partners, and friends in the industry for your continued support.
—Swipeclock Team
Posted Apr 05, 2021 - 05:39 MDT
Update
Current Status: We have now restored all data up until the time the system was taken down on April 1.
With full data restoration, all employees are now available to punch. This is a major milestone in restoring full system functionality and eventual user access.
Currently, we have multiple teams testing all systems in preparation to take the solutions online later today for payroll processing. As we bring the solution online, you may experience performance inconsistencies that are temporary.
Next Steps: We will issue an additional update later this morning at which time we hope to be better able to estimate when the full system will be available for all users. We are finalizing a guide outlining steps to prepare for the system coming back online. We will provide access to this guide in a future update.
Timeline: We will continue to provide frequent updates on our Status Page until the system is fully online: http://status.payrollservers.us/
We want to thank our customers, partners, and friends in the industry for your continued support.
—Swipeclock Team
Posted Apr 05, 2021 - 03:54 MDT
Identified
Current Status: The data recovery and restoration process is continuing as planned. We will provide a full update once the data restoration process is complete. This is the next critical step in bringing the system back online.
Next Steps: We will continue to work throughout the night toward our goal of having the system up for payroll processing tomorrow. We are finalizing a guide outlining steps to prepare for the system coming back online. Please keep in mind you may temporarily experience performance inconsistencies as we continue to bring everything online. Continue to watch our status page for updates.
Posted Apr 04, 2021 - 22:31 MDT
Update
Current Status: The data recovery and restoration process is continuing as planned.
Next Steps: We will continue to work throughout the night toward our goal of having the system up for payroll processing tomorrow. We are finalizing a guide outlining steps to prepare for the system coming back online. Please keep in mind you may temporarily experience performance inconsistencies as we continue to bring everything online. Continue to watch our status page for updates.
Posted Apr 04, 2021 - 20:55 MDT
Update
Status: On the evening of April 1, 2021, we detected suspicious activity within our cloud environment. We immediately took actions to safeguard the systems which impacted the availability of clients to access our timekeeping software. Our development team restored limited service on April 3 which returned timeclock functionality and punch collection to the TimeWorksPlus, TimeWorks, and WorkforceHub systems.
With this service restored, employees were once again able to use the WebClock and mobile app to clock in and out along with physical clocks. It also meant that punch data collected on physical clocks during the outage could be transmitted from the internal clock memory to our servers.
Next Steps: Our goal is to have the system up for payroll processing tomorrow. We are finalizing a guide outlining steps to prepare for the system coming back online. Please keep in mind you may temporarily experience performance inconsistencies as we continue to bring everything online. Watch our status page for details.
Timeline: We will provide frequent updates throughout the evening on our Status Page as we work to bring the system back online: http://status.payrollservers.us/
We appreciate your continued patience and understanding.
Thank you, —Swipeclock Team
Posted Apr 04, 2021 - 17:14 MDT
Update
Current Status: Mobile and Webclock punch collection continues to function and is performing well.
We have also restored the functionality for Webclocks embedded in third-party portals. If you use this functionality, we recommend you verify and validate.
NOTE that any TimeWorksPlus site or employee that was added to the system after March 16, 2021 is not able to punch at this time. We continue working to restore data entered into the system after March 16 and will keep you updated on these efforts.
Next Steps: All efforts remain focused on restoring full system access. We are working towards a goal of having the system available Monday morning. We will provide an update on this goal later today.
Timeline: We will issue another email update on 4/4/2021 at 7:00 pm Eastern.
We appreciate your continued patience and understanding.
Thank you, —Swipeclock Team
Posted Apr 04, 2021 - 08:14 MDT
Update
Incident Overview: On the evening of April 1, 2021, we detected suspicious activity within our cloud environment. Upon discovery, we immediately disabled access to certain systems, which impacted the availability for clients to access our time and attendance solutions. Following our recovery plan, we prioritized safely and securely restoring our systems. The next priority was the restoration of punch collection via web and mobile devices. Physical clocks have been collecting punches in offline mode.
Current Status: We are now able to report that mobile and Webclock punch collection has been restored and is now active, except in the case where Webclock is embedded in a third-party portal through an API using single sign-on. We are presently working to restore this functionality. Please direct all employees to the following link to login and punch using the Swipeclock Webclock: http://clock.payrollservers.us/
Data restoration is being executed in a staged approach. In order to restore systems as quickly as possible, we used backups with data through March 16, 2021. We are working to restore data after March 16 and will keep you updated on these efforts.
NOTE that any TimeWorksPlus site or employee that was added to the system after March 16, 2021 will not be eligible for punching at this time.
Next Steps: With punch collection now restored, all hands are focused on restoring full system access and remaining historical data. We will also be issuing guidance for partners who have embedded clocks that use Single Sign-on from a third-party system. Until this service is restored, Partners can provide clients access to http://clock.payrollservers.us/ for time tracking as long as employees have a TimeWorksPlus login.
Timeline: We will issue another email update at 4/4/2021 at 10:00 am Eastern. Resources:
We will continue to post our latest updates to our status page at: http://status.payrollservers.us/ We appreciate your continued patience and understanding. Thank you, —Swipeclock Team
Posted Apr 03, 2021 - 17:52 MDT
Update
Current Status: Swipeclock continues to work through our recovery plan steps to securely and safely restore service. To date, we have deployed next generation endpoint threat detection and response monitoring software on devices connected to our network, changed all system passwords, and continue to implement additional security controls as we work to restore service.
We are currently validating the data and going through a complete testing process. Currently, we do not have any evidence that data in our system has been compromised. We have hired independent forensic experts to investigate what occurred and what we can do to prevent this from occurring in the future. We have also asked them to confirm that no data was compromised, and we will keep you updated about any key findings.
Currently Affected Applications: TimeWorks, TimeWorks Plus, TimeWorks Mobile, TimeWorks Employee Mobile, TimeWorks Plus API, Webclock, WorkforceHub API and WorkforceHub are affected.
Fully Functioning Apps: ApplicantStack Recruit, ApplicantStack Onboard, and TimeSimplicity are not affected.
Next Steps: Upon thorough validation and testing, we will begin the process of running through the backlog of time punches collected on hardware devices and prepare to accept live punches through Webclock and the mobile apps.
We continue to work towards full system access and will continue to provide updates as we have them.
Timeline: We will issue another update later this afternoon/evening on 4/3/2021 at 7:00 pm Eastern.
Resources: Please visit our live status page for continuous updates: http://status.payrollservers.us/ We appreciate your continued patience and understanding.
Thank you,
—Swipeclock Team
Posted Apr 03, 2021 - 08:24 MDT
Update
Current Status: The team is making significant progress on restoring punch-gathering capability. This is our number one priority. However, we are still unable to confirm a time when full connectivity will be restored.
Next Steps – For Those Needing to Run Payroll: At this time, you will not be able to run payroll with punch data from Swipeclock until the systems are back online. Our team of developers is working diligently to expedite the process and we will keep you posted with any status changes.
Forensic Analysis: Currently, we do not have any evidence that data in our system has been compromised. We have hired independent forensic experts to investigate what occurred and what we can do to prevent this from occurring in the future. We have also asked them to confirm that no data was compromised, and we will keep you updated about any key findings.
Timeline: We will issue another email update on 4/3/2021 at 10:00 am Eastern.
Resources: Please continue to visit http://status.payrollservers.us/ for regular updates. We will keep the data as up-to-date as possible on that Status Page. We appreciate your continued patience and understanding.
Thank you. -Swipeclock Team
Posted Apr 02, 2021 - 15:29 MDT
Update
Current Status: We appreciate your continued patience and support as we work to restore your connectivity and bring our timekeeping systems back online as soon as possible. As our top priority, our team is working to restore punch gathering ability. We also have a team working to bring our full systems back online, but anticipate this process will take longer. The investigation into what occurred and what steps can be taken to prevent this from occurring in the future, including forensic analysis, is still ongoing.
Hardware clocks have been designed to collect punches, even in an offline mode. The data will push up to our cloud-based system once it becomes available.
Webclock and Mobile apps are unable to capture punch data at this time. We recommend instituting a manual system to track and log employee hours while we work to get these collection methods back online.
Systems Affected: TimeWorks, TimeWorks Plus, TimeWorks Mobile, TimeWorks Employee Mobile, TimeWorks Plus API, Webclock, WorkforceHub API and WorkforceHub.
Systems Online: ApplicantStack Recruit, ApplicantStack Onboard, Hub, Fileguardian, TimeSimplicity
What's Next We are currently working to rapidly restore our punch gathering solution. You can expect additional guidance around this process and what steps you might need to take by 5:00 pm Eastern.
Timeline The estimated time for our systems to be back online is still unknown.
Resources You can find a suggested email that you may send to your clients about the outage, below. This email template is also available on PEP.
As new information becomes available we will continue to keep you updated with real time information on our Status Page here: http://status.payrollservers.us
-Swipeclock Team
Posted Apr 02, 2021 - 11:32 MDT
Update
We are continuing to investigate the root cause of the timekeeping service disruption. We will provide information as it becomes available. Thank you for your patience.
Posted Apr 02, 2021 - 09:24 MDT
Update
We are currently experiencing a service disruption with TimeWorks, TimeWorks Plus, TimeWorks Mobile, TimeWorks Employee Mobile, TimeWorks Plus API, Webclock, WorkforceHub API and WorkforceHub.
ApplicantStack Recruit, ApplicantStack Onboard, TimeSimplicity are not affected.
On the evening of April 1, 2021 we detected suspicious activity within our cloud environment which impacted the ability for our customers to access our time and attendance solutions. We are both investigating the scope of the incident, while also exploring our ability to bring our punch capture ability back online as quickly as possible. We appreciate your patience and support as we work to restore connectivity and bring the systems back online. We do not currently have an estimated resolution time but will communicate updates as soon as we have them. You can also check the status for real time updates here: http://status.payrollservers.us/ We apologize for the inconvenience. Swipeclock Team
Posted Apr 02, 2021 - 07:42 MDT
Update
We are continuing to investigate the issue causing the outage to our timekeeping applications. We will post information here as it becomes available. Thank you for your patience.
Posted Apr 02, 2021 - 06:53 MDT
Update
We are continuing to investigate this issue.
Posted Apr 02, 2021 - 03:57 MDT
Investigating
We are investigating an issue with our time keeping systems including; TimeWorks, TimeWorksPlus, Webclock and WorkforceHub. These systems are currently offline. Will will post updates as information becomes available. Thank you for your patience.
Posted Apr 02, 2021 - 03:46 MDT
This incident affected: TimeWorks, TimeWorksPlus, Clocks, Webclock, TimeWorks Mobile, TimeWorksPlus API, WorkforceHUB, WorkforceHUB API, and TimeWorksPlus Employee Mobile Application.