Skip to Content

Wiki source code of Import Tool

Last modified by Rayaan Ahmed on 2026/04/07 10:25
Show last authors
1 Submitting an Import job through the Import Tool allows you to use the powerful mapping capabilities of DExTr to import data into ONEcount. Import Jobs are scheduled by the system and run every hour. Jobs run in parallel and can take from a few minutes to several hours to complete. You can enter an e-mail address into your job. If you do, an e-mail will be sent to the e-mail address when your job completes, including relevant information about errors, problems with the job, etc.
2
3 To start, choose Import/Export Tool under Data Management on the main menu:
4
5 [[image:attach:Screen Shot 2021-10-09 at 4.33.36 PM (2) copy 9.png||thumbnail="true" height="250"]]
6
7
8 You will see a list of import jobs that are already in the system. Click Create New Import/Export job in the upper-right corner to create a new import job:
9
10 [[image:attach:List of Import Jobs.png||height="250"]]
11
12 Now you'll create your import job. First, give the job a name and a brief description.
13
14 [[image:attach:Import Job.png||height="250"]]
15
16 Next, choose which DExTr mapping you will apply to this import. These are the mappings you created in the [[DExTr>>doc:OCDEV.Utilities - DExTr -- OLD]] screen.
17
18 [[image:attach:Import Job Choose Mapping.png||height="250"]]
19
20 Now choose whether the file should be accessed via FTP, SFTP, or URL. We recommend using SFTP or an HTTPS URL, since most imports involve some sort of PII. You will need to enter the full file path to your file, including the file name.
21
22 For example, if you are importing a file called CRM-data.csv, and  your file is on an SFTP site and it is in the /files/outgoing/, then in the File Path field you would enter: /files/outgoing/CRM-data.csv.
23
24 For repeat imports, some customers use a naming convention with a date place-holder in it. The macro is identified by a pair of %% markers on each end. It's definition is:
25
26 YYYY – four-digit year
27
28 YY – two-digit year
29
30 MM – two-digit month
31
32 DD – two-digit date
33
34 Using this placeholder, you can specify files such as the following:
35
36 file-%%MMYY%%.csv  or file-%%MMDDYYYY%%.csv
37
38 Please note that when using a date placeholder, ONEcount will only import a file if the date is today's date.
39
40 When using a dated naming convention, ONEcount will import files only with a matching date. In other words, if the file name is file-12252019.csv, ONEcount will only import that file on December 25, 2019. On the other hand, if the file name is file-%%MMYY%%.csv, it will import a file named file-1219.csv every day in December 2019.
41
42 After the file name, enter the e-mail address that you would like to receive a summary of the import process. You can enter a comma-separated list of addresses.
43
44 Next enter the FTP/SFTP/URL login credentials required by the remote system. ONEcount will use these credentials to fetch your file and import it.
45
46 Finally, choose the frequency for this import. Available options are Daily, Monthly, One-Time or Weekly.
47
48 [[image:attach:Import Job Frequency.png||height="250"]]
49
50 When you are done, hit Save and your Import Job will be saved into the system. It will run the next time ONEcount scans your queue for jobs, and you'll receive an e-mail notifying you when the job is completed.
51
52 === **Import Job Status** ===
53
54 When you save a job, the web UI sets it to "Active". We then have an import scheduler that pre-flights the job and moves it into the active queue. The scheduler checks to see if the SFTP credentials work, if the file is there, if the mapping is correct, etc. If the scheduler finds a problem with the file, it will update the last run time and set it to "processed." If there is nothing wrong with the file and it is ready to go, it sets the status to "Queued". It can take anywhere from a few seconds to an hour for the scheduler to get to the job and act on it. The scheduler is what sent you the e-mail complaining about the missing headers.
55
56 Once the job is queued by the scheduler, the import processor starts running the import, it changes the status to "Processing." When the import processor is done, it changes the status to "processed."
57
58 You can queue the job as many times as you want, once the import processor starts running it, the job is locked until it either finishes or until someone kills it.
59
60 The processor can take a long time to process a file~-~-we get files of 10 or 15 million records or more that can take hours to process. So while the processor is running, it updates a database with a timestamp for that specific job. It's like a heartbeat. This just lets us know the job is a live and still running properly. We then have a monitor running that checks this database and if it doesn't see a heartbeat for an hour, it assumes the job is dead. At that point, it will kill the import job and reset its status to Active so it can run again. It then dumps a copy of the import job logs and sends the team an e-mail that the job was reset.
61
62 Jobs can hang for a number of reasons. Most of the time it's transient, one of the servers it is talking to has a problem... there's various reasons. It doesn't happen too frequently, and when it does, the job processes correctly the second time and the customer doesn't notice. This doesn't happen if a job takes a long time, only if it stops processing for some reason.
63
64
65 === **Note on Source Code Handling in Import** ===
66
67 For **import transaction types**, the source code is determined as follows:
68
69 1. **Mapped Source Code in DExTr Mapper**
70 1*. If a source code is mapped in the DExTr mapper, the system will import and assign the source code based on that mapping.
71 1. **No Mapping, but Source Code Field Present**
72 1*. If no source code is mapped, but the source code field is included in the mapper:
73 1**. The import job will read the source code from the file.
74 1**. If the source code does not already exist in the system, a new source code will be created and assigned to the record.
75 1. **Null or Empty Source Code in File**
76 1*. If the source code field in the import file is null or empty:
77 1**. The system will assign the **default source code** to the transaction which is assigned on Import Tool.
78
79 === **Note on Promo Code Handling in Import** ===
80
81 For **import transaction types**, the promo code is determined as follows:
82
83 1. **Mapped Promo Code in DExTr Mapper**
84 1*. If a promo code is mapped in the DExTr mapper, the system will import and assign the promo code based on that mapping.
85 1. **No Mapping, but Promo Code Field Present**
86 1*. If no promo code is mapped, but the promo code field is included in the mapper:
87 1**. The import job will read the promo code from the file.
88 1**. If the promo code does not already exist in the system, a new promo code will be created and assigned to the record.
89 1. **Null or Empty Promo Code in File**
90 1*. If the promo code field in the import file is null or empty than that transaction will be imported with empty promo code.