ManTIME @ TempEval-3

ManTIME is an open-source machine learning pipeline for the extraction of temporal expressions from general domain texts.

Demo

This demo is using the latest version of ManTIME trained on human-annotated data only and today's date as utterance.

Lack of creativity? Use these:

"For nearly forty years, the United States has said categorically it would not tolerate totalitarian rule in its own backyard."
"I checked my diet 3 days ago. For 3 years I have been regular: I eat every day, never during the afternoon."

Download

Installation

The best way to install ManTIME on Debian-based operating systems is by using the installation script (install-mantime.sh). Thanks to Dr. Angus Roberts (University of Sheffield, UK) for the very first version of the script!

Source code

ManTIME source code is hosted by GitHub.

CRF++ models

The pipeline uses a CRFs model for the identification phase. You can use one of the following pre-trained models, depending on the training set:

Gold and Silver data: Size: 1.1Gb; MD5: 9d710523fc304cc0b8f6855c6f3b3645, download
Gold data only: Size: 179Mb; MD5: 1a936c8863056b63afb0da3f3e09a323, download
Silver data only: Size: 969Mb, MD5: c25530fffc8d28d12bf1727e606b382f, download

Related publications

Temporal expression extraction with extensive feature type selection and a posteriori label adjustment
M. Filannino, G. Nenadic
Data & Knowledge Engineering, ELSEVIER (2015)
paper, review, demo, project page, source code.

ManTIME: Temporal expression identification and normalization in TempEval-3 challenge
M. Filannino, G. Brown, G. Nenadic
Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013)
paper, poster, reviews, bibtex, demo, project page, source code.

Temporal expression normalisation in natural language texts
M. Filannino
CoRR, abs/1206.2010, 2012

Statistical validation

The following tables show the results of the model selection procedure and the analysis of statistical significant for the application of the post-processing pipeline. The figures are F1-scores.

Model selection

# shuffling	# fold	Models
# shuffling	# fold	Model 1	Model 2	Model 3	Model 4
Shuffling 0	Fold 0	83.60	83.78	84.22	83.27
	Fold 1	83.67	83.73	83.50	83.38
	Fold 2	84.38	84.78	84.44	84.49
	Fold 3	83.01	82.84	83.02	82.81
	Fold 4	83.50	83.49	83.85	83.54
	Fold 5	83.30	83.12	83.58	82.60
	Fold 6	84.29	84.44	84.77	83.82
	Fold 7	82.98	83.43	83.38	82.66
	Fold 8	81.78	82.23	82.23	81.95
	Fold 9	84.91	84.45	84.92	83.81
Shuffling 1	Fold 0	83.22	83.21	83.35	83.50
	Fold 1	82.94	83.12	83.20	82.59
	Fold 2	84.97	84.49	84.66	83.81
	Fold 3	82.85	82.38	83.07	83.28
	Fold 4	83.66	83.75	83.55	83.38
	Fold 5	84.64	84.66	84.59	84.37
	Fold 6	83.44	83.20	83.40	82.26
	Fold 7	84.54	84.12	84.83	83.56
	Fold 8	81.67	81.98	81.85	81.73
	Fold 9	83.96	83.92	83.43	83.26
Shuffling 2	Fold 0	83.20	83.66	83.22	82.78
	Fold 1	84.35	84.37	84.89	83.64
	Fold 2	83.96	83.55	84.19	83.33
	Fold 3	83.29	83.81	83.47	83.66
	Fold 4	81.88	82.00	82.05	81.42
	Fold 5	83.40	83.35	83.58	82.45
	Fold 6	83.67	83.33	83.37	83.08
	Fold 7	84.37	85.00	84.74	83.98
	Fold 8	82.87	82.64	83.02	82.89
	Fold 9	84.71	84.49	84.40	83.74
Shuffling 3	Fold 0	84.41	84.52	84.77	83.52
	Fold 1	83.42	83.08	83.28	82.50
	Fold 2	84.59	84.58	84.25	84.04
	Fold 3	83.46	84.84	83.94	83.41
	Fold 4	81.83	82.04	81.98	81.80
	Fold 5	84.66	84.21	84.66	83.64
	Fold 6	83.33	83.26	83.20	82.62
	Fold 7	82.85	82.84	82.80	82.98
	Fold 8	84.05	83.96	83.65	83.25
	Fold 9	83.72	83.99	83.54	83.06
Shuffling 4	Fold 0	82.10	82.01	82.08	81.56
	Fold 1	84.68	84.87	84.14	84.46
	Fold 2	84.62	84.60	84.66	83.78
	Fold 3	84.09	83.82	84.28	83.19
	Fold 4	83.59	83.30	83.20	82.53
	Fold 5	83.32	83.09	83.44	82.33
	Fold 6	82.94	82.94	82.61	83.07
	Fold 7	83.62	83.22	83.65	83.38
	Fold 8	84.39	84.14	84.50	83.88
	Fold 9	83.31	83.38	83.48	83.29
Mean		83.5998	83.6002	83.6576	83.1466
STD		0.8432771109	0.8259700296	0.8240431928	0.7256035067

Results of the 5x10-fold cross-validation with mean and standard deviations.

Differences among models using 5x10-fold cross-validation

Post-processing pipeline

# model	# fold	Post-processing pipeline application
# model	# fold	CRFs only	CRFs + pipeline
Model 1	Fold 0	83.60	85.43
	Fold 1	83.67	86.84
	Fold 2	84.38	86.11
	Fold 3	83.01	84.65
	Fold 4	83.50	86.15
	Fold 5	83.30	85.92
	Fold 6	84.29	86.17
	Fold 7	82.98	86.10
	Fold 8	81.78	84.98
	Fold 9	84.91	86.70
Model 2	Fold 0	83.78	85.02
	Fold 1	83.73	87.22
	Fold 2	84.78	86.29
	Fold 3	82.84	84.22
	Fold 4	83.49	85.75
	Fold 5	83.12	84.67
	Fold 6	84.44	86.34
	Fold 7	83.43	85.96
	Fold 8	82.23	85.36
	Fold 9	84.45	86.83
Model 3	Fold 0	84.22	85.34
	Fold 1	83.50	86.96
	Fold 2	84.44	85.74
	Fold 3	83.02	84.60
	Fold 4	83.85	86.42
	Fold 5	83.58	85.79
	Fold 6	84.77	86.27
	Fold 7	83.38	85.87
	Fold 8	82.23	85.39
	Fold 9	84.92	87.05
Model 4	Fold 0	83.27	85.00
	Fold 1	83.38	86.25
	Fold 2	84.49	86.31
	Fold 3	82.81	85.42
	Fold 4	83.54	86.79
	Fold 5	82.60	85.02
	Fold 6	83.82	85.80
	Fold 7	82.66	84.94
	Fold 8	81.95	84.87
	Fold 9	83.81	86.35
Mean		83.54875	85.82225
STD		0.69706	0.95457

Results of the 10-fold cross-validation with mean and standard deviations.

Differences among architectures using 10-fold cross-validation.