gccrs: Refactor lexer to handle UTF-8

gcc/rust/ChangeLog:

	* lex/rust-lex.cc (is_float_digit):Change types of param to `uint32_t`
	(is_x_digit):Likewise
	(is_octal_digit):Likewise
	(is_bin_digit):Likewise
	(check_valid_float_dot_end):Likewise
	(is_whitespace):Likewise
	(is_non_decimal_int_literal_separator):Likewise
	(is_identifier_start):Likewise
	(is_identifier_continue):Likewise
	(Lexer::skip_broken_string_input):Likewise
	(Lexer::build_token):Remove handling BOM
	(Lexer::parse_in_type_suffix):Modify use of `current_char`
	(Lexer::parse_in_decimal):Likewise
	(Lexer::parse_escape):Likewise
	(Lexer::parse_utf8_escape):Likewise
	(Lexer::parse_partial_string_continue):Likewise
	(Lexer::parse_partial_hex_escape):Likewise
	(Lexer::parse_partial_unicode_escape):Likewise
	(Lexer::parse_byte_char):Likewise
	(Lexer::parse_byte_string):Likewise
	(Lexer::parse_raw_byte_string):Likewise
	(Lexer::parse_raw_identifier):Likewise
	(Lexer::parse_non_decimal_int_literal):Likewise
	(Lexer::parse_decimal_int_or_float):Likewise
	(Lexer::peek_input):Change return type to `Codepoint`
	(Lexer::get_input_codepoint_length):Change to return 1
	(Lexer::peek_codepoint_input):Change to be wrapper of `peek_input`
	(Lexer::skip_codepoint_input):Change to be wrapper of `skip_input`
	(Lexer::test_get_input_codepoint_n_length):Deleted
	(Lexer::split_current_token):Deleted
	(Lexer::test_peek_codepoint_input):Deleted
	(Lexer::start_line):Move backwards
	(assert_source_content):New helper function for selftest
	(test_buffer_input_source):New helper function for selftest
	(test_file_input_source):Likewise
	(rust_input_source_test):New test
	* lex/rust-lex.h (rust_input_source_test):New test
	* rust-lang.cc (run_rust_tests):Add selftest

Signed-off-by: Raiki Tamura <[email protected]>

Author: tamaroning