First steps
ConFlowGen has been developed with Jupyter Notebooks in mind. While it is also possible to write your code as pure Python Scripts (see e.g. the Python Script examples), Jupyter Notebooks allow to mix the code with explanatory texts, LaTeX formulas, etc. in a single document. In addition, the generated visualisations neatly blend in.
For newcomers it is suggested to download the Anaconda distribution and first to familiarize yourself with the newly installed tools. Then, it is suggested to create a separate environment as described for the Jupyter Notebook examples. During the environment set up, ConFlowGen is automatically installed. Then, you can start JupyterLab from Anaconda
Navigator or from the CLI. Please ensure that you are in the correct conda environment (it should be conflowgen) when you work.
Prerequisites
To start, we first import ConFlowGen as a module.
[1]:
import os
import datetime
import pandas as pd
import conflowgen
Then, we set up the logger
[2]:
logger = conflowgen.setup_logger(
logging_directory="./data/logger", # use subdirectory relative to Jupyter Notebook
format_string="%(message)s" # only show log messages, discard timestamp etc.
)
Creating log directory at ./data/logger
Creating log file at ./data/logger/2022-10-23--18-19-51.log
Database selection
Now, we select a database to work in.
[3]:
database_chooser = conflowgen.DatabaseChooser(
sqlite_databases_directory="./data/db" # use subdirectory relative to Jupyter Notebook
)
demo_file_name = "my_demo.sqlite"
database_chooser.create_new_sqlite_database(demo_file_name, overwrite=True)
Creating SQLite directory at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v2.0.1/docs/notebooks/data/db
No previous database detected, creating new at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v2.0.1/docs/notebooks/data/db/my_demo.sqlite
Opening file /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v2.0.1/docs/notebooks/data/db/my_demo.sqlite
journal_mode: wal
cache_size: -32768
page_size: 4096
foreign_keys: 1
Creating new database at /home/docs/checkouts/readthedocs.org/user_builds/conflowgen/checkouts/v2.0.1/docs/notebooks/data/db/my_demo.sqlite
Creating all tables...
Seed with default values...
Number entries in table 'DeepSeaVessel': 0
Number entries in table 'Feeder': 0
Number entries in table 'Barge': 0
Number entries in table 'Train': 0
Number entries in table 'Truck': 0
Number entries in table 'Container': 0
General settings
We use
ContainerFlowGenerationManager.set_properties()
to set a name, the start_date and the end_date for the
generation process.
The start_date and end_date parameters must be provided as a datetime.date.
In our example, the end_time is set as a 21 day interval on top of the start_date
which is set to the time of execution.
Other general settings can be set here, see the ContainerFlowGenerationManager class for more information.
[4]:
today = datetime.datetime.now().date()
container_flow_generation_manager = conflowgen.ContainerFlowGenerationManager()
container_flow_generation_manager.set_properties(
name="Demo file",
start_date=today,
end_date=today + datetime.timedelta(days=21)
)
Loading destination distribution...
Creating schedules
In the next step, we add a feeder liner service to the database.
We do this by using
PortCallManager.add_vehicle().
The concrete vehicle instances are only generated when
ContainerFlowGenerationManager.generate()
is invoked.
[5]:
port_call_manager = conflowgen.PortCallManager()
At first we define a name for our new feeder liner service.
[6]:
feeder_service_name = "LX050"
By using
PortCallManager.add_vehicle(),
we can define the attributes for our feeder service.
vehicle_typedefines, that we deal with a feeder as the mode of transport. Other valid modes of transport are deep_sea_vessel, barge, or train.service_namedefines a fictional name for that service.vehicle_arrives_atspecifies the date where the first port call of this particular line is usually happening. This parameter must be adatetime.date.vehicle_arrives_at_timesets the scheduled timeslot of the port call. This parameter must be adatetime.time.average_vehicle_capacitydefines the average capacity of the vessels utilized on this line. Parameter must beintorfloat.average_moved_capacitysets the capacity which is in average moved between the feeder and the terminal at each call. Parameter must beintorfloat.next_destinationscan be set, consisting of name and frequency which can, e.g., be used as implication for storage and stacking problems. A list of name-frequency pairs is expected here.
[7]:
port_call_manager.add_vehicle(
vehicle_type=conflowgen.ModeOfTransport.feeder,
service_name=feeder_service_name,
vehicle_arrives_at=datetime.date(2021, 7, 9),
vehicle_arrives_at_time=datetime.time(11),
average_vehicle_capacity=800,
average_moved_capacity=100,
next_destinations=[
("DEBRV", 0.4), # 40% of the containers go here...
("RULED", 0.6) # and the other 60% of the containers go here.
]
)
Adding destination 'DEBRV' for schedule 'LX050' at the position 1 with a frequency of 40.00%
Adding destination 'RULED' for schedule 'LX050' at the position 2 with a frequency of 60.00%
Following the same principle and structure we can also add schedules for trains and deep sea vessels:
[8]:
port_call_manager.add_vehicle(
vehicle_type=conflowgen.ModeOfTransport.train,
service_name="JR03A",
vehicle_arrives_at=datetime.date(2021, 7, 12),
vehicle_arrives_at_time=datetime.time(17),
average_vehicle_capacity=90,
average_moved_capacity=90,
next_destinations=None # Here we don't have containers that need to be grouped by destination
)
[9]:
port_call_manager.add_vehicle(
vehicle_type=conflowgen.ModeOfTransport.deep_sea_vessel,
service_name="LX050",
vehicle_arrives_at=datetime.date(2021, 7, 10),
vehicle_arrives_at_time=datetime.time(19),
average_vehicle_capacity=16000,
average_moved_capacity=150, # for faster demo
next_destinations=[
("ZADUR", 0.3), # 30% of the containers go to ZADUR...
("CNSHG", 0.7) # and the other 70% of the containers go to CNSHG.
]
)
Adding destination 'ZADUR' for schedule 'LX050' at the position 1 with a frequency of 30.00%
Adding destination 'CNSHG' for schedule 'LX050' at the position 2 with a frequency of 70.00%
Generate the data
After we have set some exemplary schedules, we can now come to the actual generation.
The generation process of the synthetic container flow data is started with
ContainerFlowGenerationManager.generate(),
based on the information you set earlier.
With the optional parameter overwrite you determine the behavior if already some data exists - with the default behavior overwrite=True, any previous container flow data is overwritten, with overwrite=False the generation step is skipped if any data is detected.
[10]:
container_flow_generation_manager.generate()
Remove previous data...
Reloading properties and distributions...
Using transportation buffer of 0.2 when choosing the departing vehicles that adhere a schedule.
Use transport buffer of 0.2 for allocating containers delivered by trucks.
For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h
For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h
For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 684h
For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 208h
For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 208h
For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 208h
For vehicle type feeder and storage requirement empty, the container dwell times need to range from 3h to 979h
For vehicle type feeder and storage requirement standard, the container dwell times need to range from 3h to 223h
For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 3h to 223h
For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 3h to 223h
For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 3h to 878h
For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 3h to 216h
For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 3h to 216h
For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 3h to 216h
For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 691h
For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 576h
For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 576h
For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 576h
For vehicle type truck and storage requirement empty, the container dwell times need to range from 0h to 1353h
For vehicle type truck and storage requirement standard, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement reefer, the container dwell times need to range from 0h to 511h
For vehicle type truck and storage requirement dangerous_goods, the container dwell times need to range from 0h to 511h
For vehicle type train and storage requirement empty, the container dwell times need to range from 0h to 828h
For vehicle type train and storage requirement standard, the container dwell times need to range from 0h to 122h
For vehicle type train and storage requirement reefer, the container dwell times need to range from 0h to 122h
For vehicle type train and storage requirement dangerous_goods, the container dwell times need to range from 0h to 122h
For vehicle type feeder and storage requirement empty, the container dwell times need to range from 12h to 936h
For vehicle type feeder and storage requirement standard, the container dwell times need to range from 12h to 257h
For vehicle type feeder and storage requirement reefer, the container dwell times need to range from 12h to 257h
For vehicle type feeder and storage requirement dangerous_goods, the container dwell times need to range from 12h to 257h
For vehicle type deep_sea_vessel and storage requirement empty, the container dwell times need to range from 12h to 964h
For vehicle type deep_sea_vessel and storage requirement standard, the container dwell times need to range from 12h to 468h
For vehicle type deep_sea_vessel and storage requirement reefer, the container dwell times need to range from 12h to 468h
For vehicle type deep_sea_vessel and storage requirement dangerous_goods, the container dwell times need to range from 12h to 468h
For vehicle type barge and storage requirement empty, the container dwell times need to range from 0h to 532h
For vehicle type barge and storage requirement standard, the container dwell times need to range from 0h to 720h
For vehicle type barge and storage requirement reefer, the container dwell times need to range from 0h to 720h
For vehicle type barge and storage requirement dangerous_goods, the container dwell times need to range from 0h to 720h
Loading destination distribution...
Load destination distribution for service 'LX050' by feeder
Destination 'DEBRV' is frequented by 40.00% of the containers and is number 1
Destination 'RULED' is frequented by 60.00% of the containers and is number 2
Load destination distribution for service 'LX050' by deep_sea_vessel
Destination 'ZADUR' is frequented by 30.00% of the containers and is number 1
Destination 'CNSHG' is frequented by 70.00% of the containers and is number 2
Create fleet including their delivered containers for given time range for each schedule...
Create vehicles and containers for service 'LX050' of type 'feeder', progress: 1 / 3 (33.33%)
Create vehicles and containers for service 'JR03A' of type 'train', progress: 2 / 3 (66.67%)
Create vehicles and containers for service 'LX050' of type 'deep_sea_vessel', progress: 3 / 3 (100.00%)
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.
Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean: 1.89
Stddev: 0.55
(rounding errors might exist)
Free Outbound Capacity Statistics
Minimum: 90.00
Maximum: 180.00
Mean: 130.00
Stddev: 39.69
(rounding errors might exist)
Assign containers that are picked up from the terminal by a vehicle adhering a schedule to their specific vehicle instance...
Assign containers to departing vehicles that move according to a schedule...
In total, 502 containers continue their journey on a vehicle that adhere to a schedule, assigning these containers to their respective vehicles...
Progress: 1 / 502 (0.20%) containers have been assigned to a scheduled vehicle to leave the terminal again.
Progress: 502 / 502 (100.00%) containers have been assigned to a scheduled vehicle to leave the terminal again.
Containers for which no outgoing vehicle adhering to a schedule could be found: 82.67%. These will be re-assigned to another vehicle type, such as a truck.
All containers for which a departing vehicle that moves according to a schedule was available have been assigned to one.
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.
Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean: 1.89
Stddev: 0.55
(rounding errors might exist)
Free Outbound Capacity Statistics
Minimum: 0.25
Maximum: 130.50
Mean: 52.33
Stddev: 43.99
(rounding errors might exist)
Generate trucks that pick up containers...
In total 188 containers are picked up by truck, creating these trucks now...
Progress: 1 / 188 (0.53%) trucks generated to pick up containers at the terminal.
Progress: 188 / 188 (100.00%) trucks generated to pick up containers at the terminal.
All 188 trucks that pick up a container have been generated, moving 304.0 TEU.
Generate containers that are delivered by trucks...
Progress: 1 / 188 (0.53%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule
Vehicle '2' of type 'deep_sea_vessel' has no remaining capacity. The free capacity of 1.50 TEU is less than the required 2.5 TEU.
Vehicle '1' of type 'deep_sea_vessel' has no remaining capacity. The free capacity of 1.50 TEU is less than the required 2.5 TEU.
Vehicle '3' of type 'deep_sea_vessel' has no remaining capacity. The free capacity of 0.25 TEU is less than the required 2.5 TEU.
Vehicle type 'deep_sea_vessel' does not offer any capacities anymore and is thus dropped. This happened at container number 149 of 188 (i.e., at 79.26%).
Progress: 188 / 188 (100.00%) of the containers which are delivered by truck are allocated on a vehicle adhering to a schedule
All 188 containers that need to be delivered by truck have been assigned to a vehicle that adheres to a schedule, corresponding to 306.25 TEU.
Loading status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.
Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean: 1.89
Stddev: 0.55
(rounding errors might exist)
Free Outbound Capacity Statistics
Minimum: 0.25
Maximum: 90.00
Mean: 18.31
Stddev: 28.15
(rounding errors might exist)
Generate trucks that deliver containers...
In total 188 containers are delivered by truck, creating these trucks now...
Progress: 1 / 188 (0.53%) trucks generated to deliver containers to the terminal.
Progress: 188 / 188 (100.00%) trucks generated to deliver containers to the terminal.
All 188 trucks that deliver a container are created now, moving 306.25 TEU.
Assign containers to next destinations...
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type feeder, progress: 1 / 2 (50.00%)
Assign destinations to containers that leave the terminal with the service 'LX050' of the vehicle type deep_sea_vessel, progress: 2 / 2 (100.00%)
Container flow generation finished
Final capacity status of vehicles adhering to a schedule:
Use transportation buffer of 0.2 for reporting statistics.
Free Inbound Capacity Statistics
Minimum: 0.75
Maximum: 2.50
Mean: 1.89
Stddev: 0.55
(rounding errors might exist)
Free Outbound Capacity Statistics
Minimum: 0.25
Maximum: 90.00
Mean: 18.31
Stddev: 28.15
(rounding errors might exist)
Export the data
The data is exported with two lines of code.
[11]:
export_container_flow_manager = conflowgen.ExportContainerFlowManager()
path_to_exported_data = export_container_flow_manager.export("first_steps", "./data/export", overwrite=True)
Creating export folder ./data/export
Creating folder at ./data/export/first_steps
Converting SQL database into file format '.csv'
Resolving column destination of model <Model: Container>...
This is a nested call to resolve the column 'destination'
Gathering data for generating the 'deep_sea_vessels' table...
Resolving column large_scheduled_vehicle of model <Model: DeepSeaVessel>...
This is a nested call to resolve the column 'large_scheduled_vehicle'
Gathering data for generating the 'feeders' table...
Resolving column large_scheduled_vehicle of model <Model: Feeder>...
Gathering data for generating the 'barges' table...
No content found for the barges table, the file will be empty.
Gathering data for generating the 'trains' table...
Resolving column large_scheduled_vehicle of model <Model: Train>...
Resolving column truck_arrival_information_for_pickup of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_pickup'
Resolving column truck_arrival_information_for_delivery of model <Model: Truck>...
This is a nested call to resolve the column 'truck_arrival_information_for_delivery'
Saving file containers.csv
Saving file deep_sea_vessels.csv
Saving file feeders.csv
Saving file barges.csv
Saving file trains.csv
Saving file trucks.csv
Export has finished successfully.
Examining the exported data
The CSV files we can open e.g. with pandas. Here, the tabular data is presented.
[12]:
df_containers = pd.read_csv(
os.path.join(path_to_exported_data, "containers.csv"),
index_col="id",
dtype={
"delivered_by_truck": "Int64",
"picked_up_by_truck": "Int64",
"delivered_by_vehicle": "Int64",
"picked_up_by_vehicle": "Int64",
"destination_sequence_id": "Int64"
}
)
df_containers
[12]:
| weight | length | storage_requirement | delivered_by | picked_up_by_initial | picked_up_by | delivered_by_vehicle | delivered_by_truck | picked_up_by_vehicle | picked_up_by_truck | emergency_pickup | destination_sequence_id | destination_name | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | |||||||||||||
| 1 | 24 | 20 | standard | feeder | truck | truck | 1 | <NA> | <NA> | 1 | False | <NA> | NaN |
| 2 | 16 | 20 | standard | feeder | deep_sea_vessel | deep_sea_vessel | 1 | <NA> | 8 | <NA> | False | 2 | CNSHG |
| 3 | 4 | 40 | empty | feeder | deep_sea_vessel | deep_sea_vessel | 1 | <NA> | 8 | <NA> | False | 2 | CNSHG |
| 4 | 24 | 20 | standard | feeder | train | train | 1 | <NA> | 5 | <NA> | False | <NA> | NaN |
| 5 | 4 | 40 | standard | feeder | deep_sea_vessel | deep_sea_vessel | 1 | <NA> | 8 | <NA> | False | 2 | CNSHG |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 798 | 20 | 40 | standard | truck | feeder | feeder | <NA> | 372 | 3 | <NA> | False | 1 | DEBRV |
| 799 | 14 | 40 | standard | truck | feeder | feeder | <NA> | 373 | 1 | <NA> | False | 1 | DEBRV |
| 800 | 20 | 20 | reefer | truck | feeder | feeder | <NA> | 374 | 1 | <NA> | False | 2 | RULED |
| 801 | 8 | 40 | standard | truck | feeder | feeder | <NA> | 375 | 2 | <NA> | False | 1 | DEBRV |
| 802 | 4 | 40 | empty | truck | feeder | feeder | <NA> | 376 | 1 | <NA> | False | 2 | RULED |
802 rows × 13 columns
The column delivered_by contains the vehicle type a container is delivered by. For trucks, the column delivered_by_truck mentions the corresponding ID of the vehicle that is stored in the separate table trucks.csv. The same scheme applies to picked_up_by and picked_up_by_truck.
[13]:
df_trucks = pd.read_csv(
os.path.join(path_to_exported_data, "trucks.csv"),
index_col="id"
)
df_trucks
[13]:
| delivers_container | picks_up_container | realized_container_pickup_time | realized_container_delivery_time | |
|---|---|---|---|---|
| id | ||||
| 1 | False | True | 2022-11-01 17:24:47.468865 | NaN |
| 2 | False | True | 2022-11-10 09:51:58.031970 | NaN |
| 3 | False | True | 2022-11-01 07:00:38.409089 | NaN |
| 4 | False | True | 2022-11-02 06:42:00.652516 | NaN |
| 5 | False | True | 2022-11-01 14:49:36.395211 | NaN |
| ... | ... | ... | ... | ... |
| 372 | True | False | NaN | 2022-11-02 13:27:52.457890 |
| 373 | True | False | NaN | 2022-10-25 09:09:21.583387 |
| 374 | True | False | NaN | 2022-10-26 14:20:49.783157 |
| 375 | True | False | NaN | 2022-11-02 13:39:56.106383 |
| 376 | True | False | NaN | 2022-10-19 16:08:14.686009 |
376 rows × 4 columns
Here, we can see which trucks deliver a container, pick up a container, and for which time their arrivals are realized.
All vehicle types except trucks are considered vehicles that typically arrive according to a schedule and which move larger amounts of containers at once. Thus, the table has a slightly different shape. They all have ids from the same ID range and for each vehicle type the ids might be non-consecutive. So e.g. for trains, in the column delivered_by_vehicle in the container table the ID is mentioned that we can look up in the table of trains.csv.
[14]:
df_trains = pd.read_csv(
os.path.join(path_to_exported_data, "trains.csv"),
index_col="id"
)
df_trains
[14]:
| vehicle_name | capacity_in_teu | moved_capacity | realized_arrival | |
|---|---|---|---|---|
| id | ||||
| 4 | 1 | 90 | 90 | 2022-10-24 17:00:00 |
| 5 | 2 | 90 | 90 | 2022-10-31 17:00:00 |
| 6 | 3 | 90 | 90 | 2022-11-07 17:00:00 |
Corresponding CSV files exist for the other vehicles as well.