back

YOLOv2

BaseURL:
https://ml.api.utagoe.com/yolo

YOLOv2
- Dataset Structure
  - property.json
  - data
- APIs
  - train
  - stop
  - restart
  - progress
  - predict
  - output
  - list
  - delete
  - validate
  - download
Appendix
- expected workflow
- akiba's test envirionment

Dataset Structure

Pack dataset in following directory structure.

(dataset_name)
│
├── property.json
└── data
    ├── IMG_20190709_153520.jpg
    ├── IMG_20190709_153520.txt
    ├── IMG_20190709_153537.jpg
    ├── IMG_20190709_153537.txt
    ├── IMG_20190709_153542.jpg
    ├── IMG_20190709_153542.txt
    └── ...

property.json

You should prepare property.json, that have information of class label.
Example:

{
    "class_num":2,
    "classes":["car", "truck"],
    "use_tiny":false
}

class_num: number of classes.
classes: list of class label.

data

Put image files and text label files in 'data' dir.
Image and text must have same filename.

image file

Each image file must be (png or jpg or jpeg).

text label file

You should prepare text label files that have information of class label and coordinate data.
One text file include labels of one image.

Example:

0 0.371349 0.396660 0.178025 0.505219
1 0.459872 0.238579 0.234875 0.349085

In each row,

label index: for example above, 0->'car' 1->'track'
coordinate: x, y, w, h of answer bounding box. x,y: center position of bbox. w,h: width and height of bbox.

Each coordinate is in range of 0 ~ 1. this value is ratio to image size.
For example above, if picture is 100x100 px , 1st row means "its center position is (37, 39). its size is (17, 50)."

APIs

train

URL: POST /train

yolo/train create new model and new training process.
Response is url of yolo/progress.

Header

Key	Value
Content-type	application/json

Parameters

Name	Type	Required	In	Description
access_key	text	Yes	Body	your access key
model_name	text	Yes	Body	name of model
dataset	text	Yes	Body	specified dataset ID
batch_size	int	No (default=1)	Body	batch_size
subdivisions	int	No (default=1)	Body	subdivisions
lr	float	No (default=0.001)	Body	learning rate
final_epoch	int	No ¹	Body	final epoch number
source	text	No (default=None)	Body	source model name

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/train -F access_key=ExampleAccessKey1234 -F model_name=exapmle_model -F dataset=example_data -F batch_size=100 -F lr=0.001 -F source=model_0

Response

Status: 200 OK

{
    "url": "http://localhost:8000/yolo/progress?model_name=example_model"
}

stop

URL: POST /stop

yolo/stop kill training process of model.
This function is not stopping but killing. When you restart, training will start from last checkpoint.

Header

Key	Value
Content-type	application/json

Parameters

Name	Type	Required	In	Description
access_key	text	Yes	Body	your access key
model_name	text	Yes	Body	model to stop

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/stop -F access_key=ExampleAccessKey1234 -F model_name=exapmle_model

Response

Status: 200 OK

restart

URL: POST /restart

yolo/restart literally restart training process of existing stopped model.
When you restart, training will start from last checkpoint.

Header

Key	Value
Content-type	application/json

Parameters

Name	Type	Required	In	Description
access_key	text	Yes	Body	your access key
model_name	text	Yes	Body	model to restart

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/restart -F access_key=ExampleAccessKey1234 -F model_name=exapmle_model

Response

Status: 200 OK

{
    "url": "http://localhost:8000/yolo/progress?model_name=example_model"
}

progress

URL: GET /progress

yolo/progress retrieve training progress.

Header

Key	Value
Access-Key	YourAccessKey

Parameters

Name	Type	Required	In	Description
model_name	text	Yes	Query	model to get progress

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/progress --access_key ExampleAccessKey1234 --model_name exapmle_model

Response

Status: 200 OK

{
    "model_name": "example_model",
    "dataset": "example_data",
    "current_steps": 12000,
    "max_steps": 50000,
    "iou": 0.46587
}

predict

URL: POST /predict

yolo/predict run predict process.
When you predict, last checkpoint of the model is used.
This API returns URL including key for retrieve result data.

Header

Key	Value
Content-type	application/json

Parameters

Name	Type	Required	In	Description
access_key	text	Yes	Body	your access key
model_name	text	Yes	Body	model to be used
dataset_id	text	Yes²	Body	dataset id to be processed

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/predict -F access_key=ExampleAccessKey1234 -F model_name=exapmle_model -F dataset_id=example_data

Response

Status: 200 OK

{
    "url": "http://localhost:8000/yolo/output?key=20201216173126"
}

output

URL: GET /output

yolo/output retrieve result of yolo/predict.

Header

Key	Value
Access-Key	YourAccessKey

Parameters

Name	Type	Required	In	Description
key	text	Yes	Query	key specific to predict process

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/output --key 20201216173126

Response

Status: 200 OK

{
    "done": true,
    "results": [
        {
            "image": "acbde.jpg",
            "boxes": [
                {
                    "class": "vehicle",
                    "score"" 0.68,
                    "x": 0.51,
                    "y": 0.35,
                    "w": 0.23,
                    "h": 0.12
                }
            ]
        }
    ]
}

list

URL: GET /list

yolo/list retrieve list of existing models.

Header

Key	Value
Access-Key	YourAccessKey

Parameters

Name	Type	Required	In	Description

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/list --access_key ExampleAccessKey1234

Response

Status: 200 OK

{
    "done": true,
    "results": [
        {
            "image": "acbde.jpg",
            "boxes": [
                {
                    "class": "vehicle",
                    "score"" 0.68,
                    "x": 0.51,
                    "y": 0.35,
                    "w": 0.23,
                    "h": 0.12
                }
            ]
        }
    ]
}

delete

URL: POST /delete

yolo/delete deletes spcefied model.

Header

Key	Value
Content-type	application/json

Parameters

Name	Type	Required	In	Description
access_key	text	Yes	Body	your access key
model_name	text	Yes	Body	model to delete

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/delete -F access_key=ExampleAccessKey1234 -F model_name=exapmle_model

Response

Status: 200 OK

validate

URL: GET /validate

yolo/validate checks specified dataset whether its format is valid for training yolov2.

Header

Key	Value
Access-Key	YourAccessKey

Parameters

Name	Type	Required	In	Description
dataset	text	Yes	Query	dataset to validate

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/validate?dataset=example_dataset --access_key ExampleAccessKey1234

Response

Status: 200 OK

{
    "valid": false,
    "message":"......"
}

download

URL: GET /download

yolo/download provide download link of zipped model weights.

Header

Key	Value
Access-Key	YourAccessKey

Parameters

Name	Type	Required	In	Description
model_name	text	Yes	Body	model to download

Code sample

Shell

curl https://ml.api.utagoe.com/yolo/download --access_key ExampleAccessKey1234 --model_name exapmle_model

Response

Status: 200 OK

(file download)_

Appendix

expected workflow

~~By data/upload, images, labels, property.json are uploaded in DATASET_ROOT/[dataset_name]/~~
- now dataset is managed in ~/dataset(local) or minio storage.
By yolo/train, cfg, data, names, list, backups are created in MODEL_ROOT/[model_name]/
By yolo/predict, json file of result is stored at PREDICTION_ROOT/[key].json

~~DATASET_ROOT~~, MODEL_ROOT, and PREDICTION_ROOT are variable of path.

akiba's test envirionment

MODEL_ROOT = ('yolo/darknet_database/models')
~~DATASET_ROOT = ('yolo/darknet_database/datasets')~~
PREDICTION_ROOT = ('yolo/darknet_prediction') as akiba's test environment.

tree below is actual akiba's environment now.

utagoe_ml_api/yolo
│
├── admin.py
├── apps.py
├── darknet/
├── darknet_database
│   ├── datasets
│   │   └── test_bottle
│   │       ├── data
│   │       │   ├── IMG_20190709_153520.jpg
│   │       │   ├── IMG_20190709_153520.txt
│   │       │   ├── ...
│   │       │   └── ...
│   │       └── property.json    # this includes class names and other properties depend on dataset
│   └── models
│       └── test_bottle
│           ├── backup/
│           ├── test_bottle.cfg
│           ├── test_bottle.data
│           ├── test_bottle.names
│           ├── test_bottle.train.list
│           └── test_bottle.valid.list
├── darknet_prediction
│   ├── 1673578468.json
│   ├── ...
│   └── ...
├── migrations/
├── models.py
├── tests.py
├── urls.py
└── views.py

in darknet configuration, number of epoch is determined by given "max_batches" and population of the dataset.
in this API, "max_batches" is calculated by given "final_epoch" and population of the dataset. default max_batches=100000. ↩
Either data_path or dataset_id is required ↩