.png)
Exploring Machine Learning to Support Software Managers' Insights on Software Developer Productivity
Abstract
Software developer productivity is a complex issue with no single, universally accepted definition or measurement. Emerging technologies like machine learning offer a promising opportunity for more accurate productivity measurement. Semi-structured interviews were conducted to gain qualitative insights into software managers’ perception of developer productivity to identify issues and inform the development of applied machine learning solutions. It was discovered that digital distractions significantly hinder developer productivity and conventional methods to monitor developer activity were often inefficient. Therefore, machine learning models were developed to monitor developer activity by classifying screenshots captured during activity, along with the URL and text content scraped from accessed URLs. Train and test data were obtained from a cooperating software house, supplemented with online sources. For screenshot classification, transfer learning using EfficientNetV2B0 outperformed InceptionV3, Resnet50V2, and VGG16, reaching 99.6% accuracy. This was achieved without fine-tuning, which resulted in the fastest training and lowest resource consumption. For content classification, SVC hyperparameter-tuned using grid search outperformed six other classifiers, reaching 88.5% accuracy. The design concept for a web application that utilizes the developed models to help managers measure developer productivity was well-received by the managers interviewed.
Keywords
Full Text:
PDFReferences
W. A. Chapetta and G. H. Travassos, “Towards an evidence-based theoretical framework on factors influencing the software development productivity,” Empir Softw Eng, vol. 25, no. 5, pp. 3501–3543, Sep. 2020, doi: 10.1007/s10664-020-09844-5.
N. Forsgren, M.-A. Storey, C. Maddila, T. Zimmermann, B. Houck, and J. Butler, “The SPACE of Developer Productivity: There’s more to it than you think.,” Queue, vol. 19, no. 1, pp. 20–48, Feb. 2021, doi: 10.1145/3454122.3454124.
J. S. Mota, H. A. Tives, and E. D. Canedo, “Tool for Measuring Productivity in Software Development Teams,” Information, vol. 12, no. 10, p. 396, Sep. 2021, doi: 10.3390/info12100396.
J. R. Burnett and T. C. Lisk, “The Future of Employee Engagement: Real-Time Monitoring and Digital Tools for Engaging a Workforce,” International Studies of Management & Organization, vol. 49, no. 1, pp. 108–119, Jan. 2019, doi: 10.1080/00208825.2019.1565097.
S. Delaney and D. Schmidt, “A Productivity Framework for Software Development Literature Review,” in Proceedings of the 2nd International Conference on Software Engineering and Information Management, New York, NY, USA: ACM, Jan. 2019, pp. 69–74. doi: 10.1145/3305160.3305161.
E. D. Canedo and G. A. Santos, “Factors Affecting Software Development Productivity: An empirical study,” in Proceedings of the XXXIII Brazilian Symposium on Software Engineering, New York, NY, USA: ACM, Sep. 2019, pp. 307–316. doi: 10.1145/3350768.3352491.
E. Oliveira, E. Fernandes, I. Steinmacher, M. Cristo, T. Conte, and A. Garcia, “Code and commit metrics of developer productivity: a study on team leaders perceptions,” Empir Softw Eng, vol. 25, no. 4, pp. 2519–2549, Jul. 2020, doi: 10.1007/s10664-020-09820-z.
A. N. Meyer, G. C. Murphy, T. Fritz, and T. Zimmermann, “Developers’ Diverging Perceptions of Productivity,” in Rethinking Productivity in Software Engineering, 1st ed., C. Sadowski and T. Zimmermann, Eds., Berkeley, CA: Apress, 2019, pp. 137–146. doi: 10.1007/978-1-4842-4221-6_12.
L. Lavazza, S. Morasca, and D. Tosi, “An empirical study on the factors affecting software development productivity,” E-Informatica Software Engineering Journal, vol. 12, no. 1, pp. 27–49, 2018, doi: https://doi.org/10.5277/e-Inf180102.
M. Beller, V. Orgovan, S. Buja, and T. Zimmermann, “Mind the Gap: On the Relationship Between Automatically Measured and Self-Reported Productivity,” IEEE Softw, vol. 38, no. 5, pp. 24–31, Sep. 2021, doi: 10.1109/MS.2020.3048200.
T. Dohmke, M. Iansiti, and G. Richards, “Sea Change in Software Development: Economic and Productivity Analysis of the AI-Powered Developer Lifecycle,” Keystone, pp. 21–29, Jun. 2023, doi: 10.1145/3520312.3534864.
S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot,” Microsoft Research, Feb. 2023, doi: 10.48550/arxiv.2302.06590.
A. Ziegler et al., “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming, New York, NY, USA: ACM, Jun. 2022, pp. 21–29. doi: 10.1145/3520312.3534864.
J. O’Keeffe, W. Buytaert, A. Mijic, N. Brozović, and R. Sinha, “The use of semi-structured interviews for the characterisation of farmer irrigation practices,” Hydrol Earth Syst Sci, vol. 20, no. 5, pp. 1911–1924, May 2016, doi: 10.5194/hess-20-1911-2016.
E. Oliveira, T. Conte, M. Cristo, and E. Mendes, “Software Project Managers’ Perceptions of Productivity Factors,” in Proceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, New York, NY, USA: ACM, Sep. 2016, pp. 1–6. doi: 10.1145/2961111.2962626.
C. Schröer, F. Kruse, and J. M. Gómez, “A Systematic Literature Review on Applying CRISP-DM Process Model,” Procedia Comput Sci, vol. 181, pp. 526–534, Feb. 2021, doi: 10.1016/j.procs.2021.01.199.
A. Shawon, “URL Classification Dataset [DMOZ],” Kaggle. Accessed: Aug. 23, 2024. [Online]. Available: https://www.kaggle.com/datasets/shawon10/url-classification-dataset-dmoz
Y. Christian, Y.-H. Choo, N. Fazilla, and A. Yusof, “Systematic Literature Review on The Use of Machine Learning in Online Learning in the Context of Skill Achievement,” J Theor Appl Inf Technol, vol. 31, no. 6, pp. 2466–2479, Mar. 2024, [Online]. Available: www.jatit.org
P. Li, X. Rao, J. Blase, Y. Zhang, X. Chu, and C. Zhang, “CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks,” in 2021 IEEE 37th International Conference on Data Engineering (ICDE), Chania, Greece: IEEE, Apr. 2021, pp. 13–24. doi: 10.1109/ICDE51399.2021.00009.
H. Talebi and P. Milanfar, “Learning to Resize Images for Computer Vision Tasks,” in 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada: IEEE, Oct. 2021, pp. 487–496. doi: 10.1109/ICCV48922.2021.00055.
H. Herman, Y. Jaya Kumar, S. Yong Wee, and V. Kumar Perhakaran, “A Systematic Review on Deep Learning Model in Computer-aided Diagnosis for Anterior Cruciate Ligament Injury,” Curr Med Imaging, vol. 20, May 2024, doi: 10.2174/0115734056295157240418043624.
E. J. Snider, S. I. Hernandez-Torres, and R. Hennessey, “Using Ultrasound Image Augmentation and Ensemble Predictions to Prevent Machine-Learning Model Overfitting,” Diagnostics, vol. 13, no. 3, p. 417, Jan. 2023, doi: 10.3390/diagnostics13030417.
G. Kovács, “An empirical comparison and evaluation of minority oversampling techniques on a large number of imbalanced datasets,” Appl Soft Comput, vol. 83, p. 105662, Oct. 2019, doi: 10.1016/j.asoc.2019.105662.
R. Khan, Y. Qian, and S. Naeem, “Extractive based Text Summarization Using KMeans and TF-IDF,” International Journal of Information Engineering and Electronic Business, vol. 11, no. 3, pp. 33–44, May 2019, doi: 10.5815/ijieeb.2019.03.05.
Sanchana.R, Josephine Ruth Fenitha, Shanmughapriya.M, Bhavani Sree. Sk, and Nithyadevi.S, “Analysis of Twitter Data using Machine Learning Algorithms,” EPRA International Journal of Research & Development (IJRD), vol. 8, pp. 48–56, Mar. 2023, doi: 10.36713/epra12585.
V. A. Kozhevnikov and E. S. Pankratova, “Research of Text Pre-processing Methods for Preparing Data in Russian for Machine Learning,” Theoretical & Applied Science, vol. 84, no. 04, pp. 313–320, Apr. 2020, doi: 10.15863/TAS.2020.04.84.55.
H. P. Luhn, “The Automatic Creation of Literature Abstracts,” IBM J Res Dev, vol. 2, no. 2, pp. 159–165, Apr. 1958, doi: 10.1147/rd.22.0159.
A. Thakkar and K. Chaudhari, “Predicting stock trend using an integrated term frequency–inverse document frequency-based feature weight matrix with neural networks,” Applied Soft Computing Journal, vol. 96, p. 106684, Nov. 2020, doi: 10.1016/j.asoc.2020.106684.
X. Peng, Y. Bao, and Z. Huang, “Perceiving Beijing’s ‘City Image’ Across Different Groups Based on Geotagged Social Media Data,” IEEE Access, vol. 8, pp. 93868–93881, May 2020, doi: 10.1109/ACCESS.2020.2995066.
M. Tan and Q. V. Le, “EfficientNetV2: Smaller Models and Faster Training,” in Proceedings of the 38th International Conference on Machine Learning, Virtual: PMLR, Jul. 2021, pp. 10096–10106. [Online]. Available: http://arxiv.org/abs/2104.00298
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the Inception Architecture for Computer Vision,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA: IEEE, Jun. 2016, pp. 2818–2826. doi: 10.1109/CVPR.2016.308.
K. He, X. Zhang, S. Ren, and J. Sun, “Identity Mappings in Deep Residual Networks,” in 14th European Conference on Computer Vision (ECCV 2016), Amsterdam, The Netherlands: Springer, 2016, pp. 630–645. doi: 10.1007/978-3-319-46493-0_38.
K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” in 3rd International Conference on Learning Representations (ICLR 2015), San Diego, CA, USA: Computational and Biological Learning Society, Apr. 2015, pp. 1–14. [Online]. Available: http://arxiv.org/abs/1409.1556
S. Montaha et al., “BreastNet18: A High Accuracy Fine-Tuned VGG16 Model Evaluated Using Ablation Study for Diagnosing Breast Cancer from Enhanced Mammography Images,” Biology (Basel), vol. 10, no. 12, p. 1347, Dec. 2021, doi: 10.3390/biology10121347.
M. Rahimzadeh and A. Attar, “A modified deep convolutional neural network for detecting COVID-19 and pneumonia from chest X-ray images based on the concatenation of Xception and ResNet50V2,” Inform Med Unlocked, vol. 19, p. 100360, Jan. 2020, doi: 10.1016/j.imu.2020.100360.
Z. Zhao, E. B. A. Bakar, N. B. A. Razak, and M. N. Akhtar, “Corrosion image classification method based on EfficientNetV2,” Heliyon, vol. 10, no. 17, p. e36754, Sep. 2024, doi: 10.1016/j.heliyon.2024.e36754.
B. Dey, J. Ferdous, R. Ahmed, and J. Hossain, “Assessing deep convolutional neural network models and their comparative performance for automated medicinal plant identification from leaf images,” Heliyon, vol. 10, no. 1, p. e23655, Jan. 2024, doi: 10.1016/j.heliyon.2023.e23655.
N. Becherer, J. Pecarina, S. Nykl, and K. Hopkinson, “Improving optimization of convolutional neural networks through parameter fine-tuning,” Neural Comput Appl, vol. 31, no. 8, pp. 3469–3479, Aug. 2019, doi: 10.1007/s00521-017-3285-0.
M. T. R et al., “Transformative Breast Cancer Diagnosis using CNNs with Optimized ReduceLROnPlateau and Early Stopping Enhancements,” International Journal of Computational Intelligence Systems, vol. 17, no. 1, p. 14, Jan. 2024, doi: 10.1007/s44196-023-00397-1.
V. Singh, V. K. Asari, and R. Rajasekaran, “A Deep Neural Network for Early Detection and Prediction of Chronic Kidney Disease,” Diagnostics, vol. 12, no. 1, p. 116, Jan. 2022, doi: 10.3390/diagnostics12010116.
Divya and M. Narwal, “Digital Distractions in the Workplace: Exploring Cyberloafing Impact on Employee Behaviour and Innovation,” Virtual Economics, vol. 6, no. 4, pp. 7–24, Dec. 2023, doi: 10.34021/ve.2023.06.04(1).
M. Nakayama and C. C. Chen, “Digital Distractions and Remote Work,” Information Resources Management Journal, vol. 35, no. 1, pp. 1–17, Sep. 2022, doi: 10.4018/IRMJ.308675.
S. L. Vadlamani and O. Baysal, “Studying Software Developer Expertise and Contributions in Stack Overflow and GitHub,” in 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), Adelaide, SA, Australia: IEEE, Sep. 2020, pp. 312–323. doi: 10.1109/ICSME46990.2020.00038.
X. Wang, X. Lin, and N. Hajli, “Understanding Software Engineers’ Skill Development in Software Development,” Journal of Computer Information Systems, vol. 61, no. 2, pp. 108–117, Mar. 2021, doi: 10.1080/08874417.2019.1566805.
S. Janssens and V. Zaytsev, “Go with the flow: software engineers and distractions,” in Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, New York, NY, USA: ACM, Oct. 2022, pp. 934–938. doi: 10.1145/3550356.3559101.
V. W.-S. Tseng, M. L. Lee, L. Denoue, and D. Avrahami, “Overcoming Distractions during Transitions from Break to Work using a Conversational Website-Blocking System,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, New York, NY, USA: ACM, May 2019, pp. 1–13. doi: 10.1145/3290605.3300697.
L. Song, “An AI-Powered System to Increase User’s Productivity using Computer Vision and Machine Learning,” in Computer Science & Information Technology (CS & IT), Academy & Industry Research Collaboration Center, May 2024, pp. 183–192. doi: 10.5121/csit.2024.141016.
D. Rogers, A. Preece, M. Innes, and I. Spasic, “Real-Time Text Classification of User-Generated Content on Social Media: Systematic Review,” IEEE Trans Comput Soc Syst, vol. 9, no. 4, pp. 1154–1166, Aug. 2022, doi: 10.1109/TCSS.2021.3120138.
DOI: https://doi.org/10.47738/jads.v6i3.661
Refbacks
- There are currently no refbacks.
Journal of Applied Data Sciences
| ISSN | : | 2723-6471 (Online) |
| Publisher | : | Bright Publisher |
| Website | : | http://bright-journal.org/JADS |
| : | taqwa@amikompurwokerto.ac.id (principal contact) | |
| support@bright-journal.org (technical issues) |
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0