gccrs: Refactor lexer to handle UTF-8

gcc/rust/ChangeLog:

	* lex/rust-lex.cc (is_float_digit):
	(is_x_digit):
	(is_octal_digit):
	(is_bin_digit):
	(check_valid_float_dot_end):
	(is_whitespace):
	(is_non_decimal_int_literal_separator):
	(is_identifier_start):
	(is_identifier_continue):
	(Lexer::peek_input):
	(Lexer::build_token):
	(Lexer::parse_in_type_suffix):
	(Lexer::parse_in_decimal):
	(Lexer::parse_escape):
	(Lexer::parse_utf8_escape):
	(Lexer::parse_partial_string_continue):
	(Lexer::parse_partial_hex_escape):
	(Lexer::parse_partial_unicode_escape):
	(Lexer::parse_byte_char):
	(Lexer::parse_byte_string):
	(Lexer::parse_raw_byte_string):
	(Lexer::parse_raw_identifier):
	(Lexer::skip_broken_string_input):
	(Lexer::parse_non_decimal_int_literal):
	(Lexer::parse_decimal_int_or_float):
	(Lexer::get_input_codepoint_length):
	(Lexer::peek_codepoint_input):
	(Lexer::test_get_input_codepoint_n_length):
	(Lexer::test_peek_codepoint_input):
	* lex/rust-lex.h:

gcc/rust/ChangeLog:

	* lex/rust-lex.cc (is_float_digit):
	(is_x_digit):
	(is_octal_digit):
	(is_bin_digit):
	(check_valid_float_dot_end):
	(is_whitespace):
	(is_non_decimal_int_literal_separator):
	(is_identifier_start):
	(is_identifier_continue):
	(Lexer::peek_input):
	(Lexer::build_token):
	(Lexer::parse_in_type_suffix):
	(Lexer::parse_in_decimal):
	(Lexer::parse_escape):
	(Lexer::parse_utf8_escape):
	(Lexer::parse_partial_string_continue):
	(Lexer::parse_partial_hex_escape):
	(Lexer::parse_partial_unicode_escape):
	(Lexer::parse_byte_char):
	(Lexer::parse_byte_string):
	(Lexer::parse_raw_byte_string):
	(Lexer::parse_raw_identifier):
	(Lexer::skip_broken_string_input):
	(Lexer::parse_non_decimal_int_literal):
	(Lexer::parse_decimal_int_or_float):
	(Lexer::get_input_codepoint_length):
	(Lexer::peek_codepoint_input):
	(Lexer::test_get_input_codepoint_n_length):
	(Lexer::test_peek_codepoint_input):
	* lex/rust-lex.h:

gcc/rust/ChangeLog:

	* lex/rust-lex.cc (is_float_digit):
	(is_x_digit):
	(is_octal_digit):
	(is_bin_digit):
	(check_valid_float_dot_end):
	(is_whitespace):
	(is_non_decimal_int_literal_separator):
	(is_identifier_start):
	(is_identifier_continue):
	(Lexer::peek_input):
	(Lexer::build_token):
	(Lexer::parse_in_type_suffix):
	(Lexer::parse_in_decimal):
	(Lexer::parse_escape):
	(Lexer::parse_utf8_escape):
	(Lexer::parse_partial_string_continue):
	(Lexer::parse_partial_hex_escape):
	(Lexer::parse_partial_unicode_escape):
	(Lexer::parse_byte_char):
	(Lexer::parse_byte_string):
	(Lexer::parse_raw_byte_string):
	(Lexer::parse_raw_identifier):
	(Lexer::skip_broken_string_input):
	(Lexer::parse_non_decimal_int_literal):
	(Lexer::parse_decimal_int_or_float):
	(Lexer::get_input_codepoint_length):
	(Lexer::peek_codepoint_input):
	(Lexer::test_get_input_codepoint_n_length):
	(Lexer::test_peek_codepoint_input):
	(assert_source_content):
	(test_buffer_input_source):
	(rust_input_source_test):
	* lex/rust-lex.h (rust_input_source_test):
	* rust-lang.cc (run_rust_tests):

Author: tamaroning