Changes for page Import Tool

Last modified by Admin User on 2024/10/03 18:28
From version 2.1
edited by Admin
on 2020/03/23 19:06
Change comment: There is no comment for this version
To version 4.1
edited by Admin User
on 2024/10/03 18:28
Change comment: There is no comment for this version

Summary

Details

Page properties
Parent
... ... @@ -1,0 +1,1 @@
1 +System Utilities
Author
... ... @@ -1,1 +1,1 @@
1 -XWiki.Admin
1 +XWiki.admin
Content
... ... @@ -1,10 +1,9 @@
1 1  Submitting an Import job through the Import Tool allows you to use the powerful mapping capabilities of DExTr to import data into ONEcount. Import Jobs are scheduled by the system and run every hour. Jobs run in parallel and can take from a few minutes to several hours to complete. You can enter an e-mail address into your job. If you do, an e-mail will be sent to the e-mail address when your job completes, including relevant information about errors, problems with the job, etc.
2 2  
3 -To start, choose Import/Export Tool under Utilities on the main menu:
3 +To start, choose Import/Export Tool under Data Management on the main menu:
4 4  
5 -[[image:attach:Import Tool Nav.png||width="800"]]
5 +[[image:attach:Screen Shot 2021-10-09 at 4.33.36 PM (2) copy 9.png||thumbnail="true" height="250"]]
6 6  
7 -\\
8 8  
9 9  You will see a list of import jobs that are already in the system. Click Create New Import/Export job in the upper-right corner to create a new import job:
10 10  
... ... @@ -50,8 +50,16 @@
50 50  
51 51  When you are done, hit Save and your Import Job will be saved into the system. It will run the next time ONEcount scans your queue for jobs, and you'll receive an e-mail notifying you when the job is completed.
52 52  
53 -\\
52 +=== **Import Job Status** ===
54 54  
55 -\\
54 +When you save a job, the web UI sets it to "Active". We then have an import scheduler that pre-flights the job and moves it into the active queue. The scheduler checks to see if the SFTP credentials work, if the file is there, if the mapping is correct, etc. If the scheduler finds a problem with the file, it will update the last run time and set it to "processed." If there is nothing wrong with the file and it is ready to go, it sets the status to "Queued". It can take anywhere from a few seconds to an hour for the scheduler to get to the job and act on it. The scheduler is what sent you the e-mail complaining about the missing headers.
56 56  
57 -\\
56 +Once the job is queued by the scheduler, the import processor starts running the import, it changes the status to "Processing." When the import processor is done, it changes the status to "processed."
57 +
58 +You can queue the job as many times as you want, once the import processor starts running it, the job is locked until it either finishes or until someone kills it.
59 +
60 +The processor can take a long time to process a file~-~-we get files of 10 or 15 million records or more that can take hours to process. So while the processor is running, it updates a database with a timestamp for that specific job. It's like a heartbeat. This just lets us know the job is a live and still running properly. We then have a monitor running that checks this database and if it doesn't see a heartbeat for an hour, it assumes the job is dead. At that point, it will kill the import job and reset its status to Active so it can run again. It then dumps a copy of the import job logs and sends the team an e-mail that the job was reset.
61 +
62 +Jobs can hang for a number of reasons. Most of the time it's transient, one of the servers it is talking to has a problem... there's various reasons. It doesn't happen too frequently, and when it does, the job processes correctly the second time and the customer doesn't notice. This doesn't happen if a job takes a long time, only if it stops processing for some reason.
63 +
64 +
Import Job Choose Mapping.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +112.4 KB
Content
Import Job Frequency.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +111.0 KB
Content
Import Job.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +106.0 KB
Content
Import Tool Nav.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +84.2 KB
Content
List of Import Jobs.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +90.7 KB
Content
Screen Shot 2021-10-09 at 4.33.36 PM (2) copy 9.png
Author
... ... @@ -1,0 +1,1 @@
1 +XWiki.XWikiGuest
Size
... ... @@ -1,0 +1,1 @@
1 +17.4 KB
Content
Confluence.Code.ConfluencePageClass[0]
id
... ... @@ -1,1 +1,1 @@
1 -62521423
1 +23331842
url
... ... @@ -1,1 +1,1 @@
1 -https://info.onecount.net//wiki/spaces/OD/pages/62521423/Import Tool
1 +https://info.onecount.net//wiki/spaces/OD/pages/23331842/Import Tool