Page 44 - SMILESENG

Intl. Summer School on Search- and Machine Learning-based Software Engineering
destructured into the arrival time (c.timestamp), arrival floor (c.source), and destination floor (c.destination).
Unfortunately, this solution is not ideal because it cannot handle element collections, such as the lists of elevator po- sitions (E ) and passenger calls (C ) from our elevation case study [7]. The current workaround is to allow user-defined functions to reduce the test inputs into domain-specific boolean or numeric features, which can then be used to generate the MRs [5]. For instance, T Dworst(E1, E2) is a domain-specific function which computes the distance between the elevator positions E1 and E2 [7].
Our proposed solution, inspired by lambda calculus [9], is to add higher-order functions such as fold, map and filter to GASSERTMRS’s expression language to enable it to reason on item sequences such as lists or sets. Implementing this feature implies introducing the new sequence and function types for expressions. The new function expressions could use the following literal syntax:
[param1, param2, ...]{expression}
With this extension, all of the user-defined functions we use for the elevation case study can be generated by GASSERTMRS, meaning that there would be no need for the users to define them. For example, TDworst(E1,E2) could be expressed by GASSERTMRS as follows:
sum(map([e1, e2]{abs(e1 − e2)}, zip(E1, E2)))
here, zip(C1, C2) combines pairs of elements from C1 and C2 into a single sequence, map(F,C) applies the transformation F to each of the elements from C, and sum(C) computes the sum of all the elements from C. All of these are generic functions which could be part of GASSERTMRS’s expression language. The benefit of this extension is two-fold: Firstly, GASSERTMRS becomes more expressive and generic, and secondly, the implicit bias introduced by user-defined func- tions is eliminated.
IV. PRELIMINARY RESULTS
We evaluated GASSERTMRS (with no support for higher- order functions) on ORONA’s case study [5], comparing the automatically generated MRs with ones that were manually generated with the assistance from domain experts [7].
The same test suite and system variants with manually seeded faults were used to evaluate both approaches. The selected Metamorphic Relation Input Patterns (MRIPs), which define the input relations of the MRs, were the following:
• MRIP1: An additional passenger call is appended.
• MRIP2: One or more additional elevators are enabled.
• MRIP3: The initial position of the elevators is changed. Table I shows the results obtained by GASSERTMRS
compared with the manually generated MRs. This evaluation shows that GASSERTMRS is able to generate non-trivial MRs which dominate the results obtained by the manual MRs in terms of both detected failure count and mutation score. Unlike the manual MRs, the ones generated by GASSERTMRS did have some FPs, but the median was always 0.
TABLE I
EVALUATION RESULTS (MEDIAN) [5]
MRIP Metric Operator AWT ≥
Detected Failures Mutation Score GAssertMRs Manual GAssertMRs Manual
≤ MRIP1 TD ≥
16.5 13.0 28.0 14.0 12.0 12.5 30.5 4.5 20.0 7.5
9.0 0.0
11.0 0.0
74.0 74.0 9.0 0.5 47.5 8.5 2.0 1.5 22.0 2.0 28.5 1.5 50.0 0.0 58.0 1.5 30.0 0.5 16.0 0.5 19.0 0.0
11.80% 14.61% 12.36% 13.48% 16.85%
4.49%
7.87%
11.24% 14.61% 13.48%
5.06% 7.87% 0.00% 0.00%
TM AWT
MRIP2 TD TM
≤ ≥ ≤ ≥ ≤ ≥ ≤ ≥ ≤
19.10% 18.54%
AWT ≥ ≤
MRIP3 TD ≥ ≤
TM ≥ ≤
8.43%
12.36% 1.12% 8.99%
13.48% 19.10% 17.98% 12.36%
7.87% 2.25%
0.56%
9.55% 1.12% 2.25% 1.12% 0.00% 1.69% 0.56% 0.56% 0.00%
32
In conclusion, GASSERTMRS appears to be a feasible approach for generating effective MRs instead of manually defining them.
V. CONCLUSION
This paper summarizes our approach to automatically gener- ate MRs for CPSs by using correct/incorrect execution samples and genetic programming. We also describe our experience of using our implementation of this approach, GASSERTMRS, on an industrial case study from the elevation domain provided be ORONA. Furthermore, we point out the current limitations that hinder the general adoption of this technique and present the solutions which we intend to implement.
REFERENCES
[1] R. Baheti and H. Gill, “Cyber-physical systems,” The impact of control technology, vol. 12, no. 1, pp. 161–166, 2011.
[2] A. Kane, T. Fuhrman, and P. Koopman, “Monitor based oracles for cyber-physical system testing: Practical experience report,” in 2014 44th Annual IEEE/IFIP International Conference on Dependable Systems and Networks. IEEE, 2014, pp. 148–155.
[3] E.T.Barr,M.Harman,P.McMinn,M.Shahbaz,andS.Yoo,“Theoracle problem in software testing: A survey,” IEEE TSE, vol. 41, no. 5, pp. 507–525, 2014.
[4] Orona, “Orona group,” https://www.orona-group.com/, 2021.
[5] J. Ayerdi, V. Terragni, A. Arrieta, P. Tonella, G. Sagardui, and M. Arratibel, “Generating metamorphic relations for cyber-physical systems with genetic programming: an industrial case study,” in Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE 2021, 2021, pp. 1264–1274. [Online].
Available: https://doi.org/10.1145/3468264.3473920
[6] T. Y. Chen, S. C. Cheung, and S. M. Yiu, “Metamorphic testing: A
new approach for generating next test cases,” Technical Report HKUST- CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology, Tech. Rep., 1998.
[7] J. Ayerdi, S. Segura, A. Arrieta, G. Sagardui, and M. Arratibel, “Qos- aware metamorphic testing: An elevation case study,” in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2020.
[8] V. Terragni, G. Jahangirova, P. Tonella, and M. Pezze`, “Evolutionary improvement of assertion oracles,” in ESEC/FSE, 2020, pp. 1178–1189.
[9] H.P.Barendregtetal.,Thelambdacalculus. North-HollandAmsterdam,
1984, vol. 3.